Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 709932 Details for
Bug 921462
[TEXT][engine] CancellationException is logged as ERROR even though it is expected
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
engine logs
engine.log (text/x-log), 7.92 MB, created by
Gadi Ickowicz
on 2013-03-14 09:24:01 UTC
(
hide
)
Description:
engine logs
Filename:
MIME Type:
Creator:
Gadi Ickowicz
Created:
2013-03-14 09:24:01 UTC
Size:
7.92 MB
patch
obsolete
>2013-02-28 13:37:01,999 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 35) Start time: 2/28/13 1:37 PM >2013-02-28 13:37:02,682 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ServerService Thread Pool -- 35) Failed to decrypt Data must start with zero >2013-02-28 13:37:02,682 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ServerService Thread Pool -- 35) Failed to decrypt value for property TruststorePass will be used encrypted value >2013-02-28 13:37:02,773 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 35) VDSBrokerFrontend: 2/28/13 1:37 PM >2013-02-28 13:37:02,781 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 35) CpuFlagsManager: 2/28/13 1:37 PM >2013-02-28 13:37:02,783 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 35) Setting audit clean up manager to run at: 35 35 3 * * ? >2013-02-28 13:37:02,812 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 35) AuditLogCleanupManager: 2/28/13 1:37 PM >2013-02-28 13:37:02,819 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 35) TagsDirector initialization >2013-02-28 13:37:02,845 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 35) Tag root added to tree >2013-02-28 13:37:02,858 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 35) TagsDirector: 2/28/13 1:37 PM >2013-02-28 13:37:02,866 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 35) IsoDomainListSyncronizer: 2/28/13 1:37 PM >2013-02-28 13:37:02,896 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 35) InitHandlers: 2/28/13 1:37 PM >2013-02-28 13:37:02,915 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 35) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2013-02-28 13:37:02,921 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 35) ErrorTranslator: 2/28/13 1:37 PM >2013-02-28 13:37:02,922 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 35) VdsErrorTranslator: 2/28/13 1:37 PM >2013-02-28 13:37:02,923 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 35) Mark uncompleted jobs as UNKNOWN: 2/28/13 1:37 PM >2013-02-28 13:37:02,974 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 35) JobRepositoryCleanupManager: 2/28/13 1:37 PM >2013-02-28 13:37:02,976 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 35) AutoRecoveryManager: 2/28/13 1:37 PM >2013-02-28 13:37:02,981 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 35) ExecutionMessageDirector: 2/28/13 1:37 PM >2013-02-28 13:37:03,017 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 36) InitResourceManager: 2/28/13 1:37 PM >2013-02-28 13:37:03,019 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 36) ResourceManager::ResourceManager::Entered >2013-02-28 13:37:03,027 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (ServerService Thread Pool -- 33) UsersDomainsCacheManager: 2/28/13 1:37 PM >2013-02-28 13:37:03,098 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (ServerService Thread Pool -- 33) DbUserCacheManager: 2/28/13 1:37 PM >2013-02-28 13:37:03,142 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-1) Start refreshing all users data >2013-02-28 13:37:03,228 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ServerService Thread Pool -- 36) AsyncTaskManager: Initialization of AsyncTaskManager completed successfully. >2013-02-28 13:37:03,228 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 36) AsyncTaskManager: 2/28/13 1:37 PM >2013-02-28 13:37:03,237 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (ServerService Thread Pool -- 36) Initialization of OvfDataUpdater completed successfully. >2013-02-28 13:37:03,238 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 36) OvfDataUpdater: 2/28/13 1:37 PM >2013-02-28 13:37:03,240 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 36) VdsLoadBalancer: 2/28/13 1:37 PM >2013-02-28 13:37:03,244 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (pool-3-thread-1) MacPoolManager started: 2/28/13 1:37 PM >2013-02-28 13:37:03,268 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 36) Init VM Custom Properties utilities: 2/28/13 1:37 PM >2013-02-28 13:37:03,415 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (pool-3-thread-1) MacPoolManager finished: 2/28/13 1:37 PM >2013-02-28 13:39:58,724 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Loaded file "/usr/share/ovirt-engine/conf/engine.conf.defaults". >2013-02-28 13:39:58,724 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Loaded file "/etc/sysconfig/ovirt-engine". >2013-02-28 13:39:58,725 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine". >2013-02-28 13:39:58,725 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_PROPERTIES" is "". >2013-02-28 13:39:58,725 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2013-02-28 13:39:58,725 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_PERM_MIN" is "256m". >2013-02-28 13:39:58,725 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "JBOSS_HOME" is "/usr/share/jbossas". >2013-02-28 13:39:58,725 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_DB_USER" is "engine". >2013-02-28 13:39:58,725 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2013-02-28 13:39:58,726 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2013-02-28 13:39:58,726 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_STOP_TIME" is "10". >2013-02-28 13:39:58,726 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_FQDN" is "gadi-rhevm.scl.lab.tlv.redhat.com". >2013-02-28 13:39:58,726 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_HEAP_MAX" is "1g". >2013-02-28 13:39:58,726 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2013-02-28 13:39:58,726 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_LOCK" is "/var/lock/ovirt-engine". >2013-02-28 13:39:58,726 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_STOP_INTERVAL" is "1". >2013-02-28 13:39:58,726 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_PROXY_ENABLED" is "true". >2013-02-28 13:39:58,726 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2013-02-28 13:39:58,727 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_APPS" is "engine.ear". >2013-02-28 13:39:58,727 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_DB_PASSWORD" is "64c5fd2979a86168". >2013-02-28 13:39:58,727 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2013-02-28 13:39:58,727 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2013-02-28 13:39:58,727 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2013-02-28 13:39:58,727 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_HTTP_PORT" is "8700". >2013-02-28 13:39:58,727 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_AJP_PORT" is "8702". >2013-02-28 13:39:58,727 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_PERM_MAX" is "256m". >2013-02-28 13:39:58,727 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_VERBOSE_GC" is "false". >2013-02-28 13:39:58,728 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_HTTPS_PORT" is "8701". >2013-02-28 13:39:58,728 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2013-02-28 13:39:58,728 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2013-02-28 13:39:58,728 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_NOFILE" is "65535". >2013-02-28 13:39:58,728 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2013-02-28 13:39:58,728 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2013-02-28 13:39:58,728 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_PID" is "/var/run/ovirt-engine.pid". >2013-02-28 13:39:58,728 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2013-02-28 13:39:58,728 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_HTTP_ENABLED" is "false". >2013-02-28 13:39:58,728 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_GROUP" is "ovirt". >2013-02-28 13:39:58,729 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "JAVA_HOME" is "/usr/lib/jvm/jre-1.7.0-openjdk.x86_64". >2013-02-28 13:39:58,729 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2013-02-28 13:39:58,729 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_DB_SSL" is "false". >2013-02-28 13:39:58,729 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2013-02-28 13:39:58,729 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_HEAP_MIN" is "1g". >2013-02-28 13:39:58,729 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_AJP_ENABLED" is "true". >2013-02-28 13:39:58,729 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-1) Value of property "ENGINE_USER" is "ovirt". >2013-02-28 13:52:19,604 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 13:52:19,678 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) AuditLogType: UNASSIGNED not exist in string table >2013-02-28 13:52:19,678 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2013-02-28 13:52:19,679 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2013-02-28 13:52:19,679 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2013-02-28 13:52:19,679 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) AuditLogType: DWH_STOPPED not exist in string table >2013-02-28 13:52:19,679 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) AuditLogType: DWH_STARTED not exist in string table >2013-02-28 13:52:19,679 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) AuditLogType: DWH_ERROR not exist in string table >2013-02-28 13:52:19,679 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) AuditLogType: EXTERNAL_EVENT_NORMAL not exist in string table >2013-02-28 13:52:19,680 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) AuditLogType: EXTERNAL_EVENT_WARNING not exist in string table >2013-02-28 13:52:19,680 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) AuditLogType: EXTERNAL_EVENT_ERROR not exist in string table >2013-02-28 13:52:19,680 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) AuditLogType: EXTERNAL_ALERT not exist in string table >2013-02-28 13:52:19,688 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) AuditLogType: USER_ATTACH_DISK_TO_VM not have severity. Assumed Normal >2013-02-28 13:52:19,689 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) AuditLogType: USER_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2013-02-28 13:52:19,689 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) AuditLogType: USER_FAILED_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2013-02-28 13:52:19,689 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not have severity. Assumed Normal >2013-02-28 13:52:19,689 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) AuditLogType: VM_MIGRATION_START_SYSTEM_INITIATED not have severity. Assumed Normal >2013-02-28 13:52:19,692 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not have severity. Assumed Normal >2013-02-28 13:52:19,733 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:52:59,085 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 13:52:59,136 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:03,560 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:03,605 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:04,424 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:04,453 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:04,599 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:04,622 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:04,625 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-02-28 13:53:04,662 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 13:53:06,856 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:06,873 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:06,961 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:06,985 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:07,081 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:07,091 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:07,243 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:07,254 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:07,344 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:07,365 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:07,438 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:07,457 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:07,522 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:07,538 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:07,629 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:07,641 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:07,732 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:07,751 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:07,822 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:07,831 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:07,908 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:07,933 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:08,013 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:08,040 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:08,124 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:08,136 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:08,231 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:08,243 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:08,307 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:08,316 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:08,399 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:08,408 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:08,496 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:08,510 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:08,577 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:08,588 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:08,696 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:08,707 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:08,788 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:09,025 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:09,105 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:09,115 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:09,201 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:09,225 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:09,301 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:09,327 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:09,413 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:09,434 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:09,506 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:09,515 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:09,601 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:09,610 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:09,698 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:09,710 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:09,787 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:09,797 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:09,914 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:09,927 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:10,011 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:10,019 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:10,093 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:10,105 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:10,178 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:10,192 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:10,268 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:10,292 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:10,582 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:10,598 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:10,669 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:10,681 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:10,764 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:10,784 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:10,862 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:10,891 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:10,973 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:10,988 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:11,059 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:11,068 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:11,137 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:11,156 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:11,299 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:11,322 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:11,404 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:11,419 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:11,488 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:11,498 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:11,577 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:11,600 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:11,682 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:11,710 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:11,785 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:11,807 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:11,885 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:11,907 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:11,980 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:12,002 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:12,083 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:12,106 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:12,197 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:12,222 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:12,479 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:12,499 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:13,024 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:13,037 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:13,449 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-3) [6787c30d] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-02-28 13:53:13,842 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:13,865 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:13,938 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:13,965 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:14,030 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:14,041 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:14,118 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:14,136 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:14,743 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:14,762 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:15,088 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-7) [719360db] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: adf1b443-62a2-486f-ad51-693efa6441c8 Type: StoragePool >2013-02-28 13:53:15,499 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:15,515 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:15,574 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:15,584 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:15,934 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:15,950 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:16,579 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:16,608 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:16,672 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:16,693 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:17,731 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-4) [12ea84ee] Running command: AddVdsCommand internal: false. Entities affected : ID: 78382612-3f27-4ea7-960c-f512fecd553c Type: VdsGroups >2013-02-28 13:53:17,831 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-4) [5dfd03f4] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: a0688005-fabc-49e6-8d53-e27e40ce06de Type: VDS >2013-02-28 13:53:17,849 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-4) [5dfd03f4] Lock freed to object EngineLock [exclusiveLocks= key: adf1b443-62a2-486f-ad51-693efa6441c8 value: REGISTER_VDS >, sharedLocks= ] >2013-02-28 13:53:17,854 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-4) [5dfd03f4] Cannot get vdsManager for vdsid=a0688005-fabc-49e6-8d53-e27e40ce06de >2013-02-28 13:53:17,858 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [5dfd03f4] START, RemoveVdsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = a0688005-fabc-49e6-8d53-e27e40ce06de), log id: 6ca38ed6 >2013-02-28 13:53:17,858 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-4) [5dfd03f4] Cannot get vdsManager for vdsid=a0688005-fabc-49e6-8d53-e27e40ce06de >2013-02-28 13:53:17,859 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [5dfd03f4] FINISH, RemoveVdsVDSCommand, log id: 6ca38ed6 >2013-02-28 13:53:17,860 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-4) [5dfd03f4] Cannot get vdsManager for vdsid=a0688005-fabc-49e6-8d53-e27e40ce06de >2013-02-28 13:53:17,862 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [5dfd03f4] START, AddVdsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = a0688005-fabc-49e6-8d53-e27e40ce06de), log id: 3b5ce0d4 >2013-02-28 13:53:17,862 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [5dfd03f4] AddVds - entered , starting logic to add VDS a0688005-fabc-49e6-8d53-e27e40ce06de >2013-02-28 13:53:17,864 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [5dfd03f4] AddVds - VDS a0688005-fabc-49e6-8d53-e27e40ce06de was added, will try to add it to the resource manager >2013-02-28 13:53:17,876 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-4) [5dfd03f4] Eneterd VdsManager:constructor >2013-02-28 13:53:17,876 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-4) [5dfd03f4] vdsBroker(10.35.160.152,54,321) >2013-02-28 13:53:17,959 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-4) [5dfd03f4] ResourceManager::AddVds - VDS a0688005-fabc-49e6-8d53-e27e40ce06de was added to the Resource Manager >2013-02-28 13:53:17,959 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [5dfd03f4] FINISH, AddVdsVDSCommand, log id: 3b5ce0d4 >2013-02-28 13:53:18,042 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-50) [4c6ece9f] Running command: InstallVdsCommand internal: true. Entities affected : ID: a0688005-fabc-49e6-8d53-e27e40ce06de Type: VDS >2013-02-28 13:53:18,042 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-50) [4c6ece9f] Before Installation host a0688005-fabc-49e6-8d53-e27e40ce06de, cheetah01.scl.lab.tlv.redhat.com >2013-02-28 13:53:18,190 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-3-thread-50) [4c6ece9f] Installation 10.35.160.152: Connected to host 10.35.160.152 with SSH key fingerprint: 13:fc:73:f1:4a:7c:f5:6a:13:3c:b9:e2:50:10:73:7f >2013-02-28 13:53:18,237 INFO [org.ovirt.engine.core.bll.VdsDeploy] (pool-3-thread-50) [4c6ece9f] Installation of 10.35.160.152. Executing command via SSH umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && python -c "import sys, tarfile; tarfile.open(fileobj=sys.stdin, mode='r|').extractall(path='${MYTMP}')" && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2013-02-28 13:53:18,238 INFO [org.ovirt.engine.core.utils.archivers.tar.CachedTar] (pool-3-thread-50) Tarball '/var/cache/ovirt-engine/ovirt-host-deploy.tar' is missing, creating >2013-02-28 13:53:18,536 INFO [org.ovirt.engine.core.utils.ssh.SSHDialog] (pool-3-thread-50) SSH execute root@10.35.160.152 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && python -c "import sys, tarfile; tarfile.open(fileobj=sys.stdin, mode='r|').extractall(path='${MYTMP}')" && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2013-02-28 13:53:18,701 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:18,709 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:18,798 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:18,818 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:18,905 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:18,918 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:19,260 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Initializing >2013-02-28 13:53:19,291 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Environment setup >2013-02-28 13:53:19,307 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Installation packages setup >2013-02-28 13:53:20,966 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-52) Initializing Host: cheetah01.scl.lab.tlv.redhat.com >2013-02-28 13:53:21,371 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Programs detection >2013-02-28 13:53:21,384 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Environment customization >2013-02-28 13:53:21,401 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Logs at host located at: '/tmp/ovirt-host-deploy-20130228135321.log' >2013-02-28 13:53:21,440 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host 10.35.160.152 reports unique id 75f723b4-e40e-4955-b7f1-ab62144e9278 >2013-02-28 13:53:21,482 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 75f723b4-e40e-4955-b7f1-ab62144e9278 to Host 10.35.160.152 >2013-02-28 13:53:21,541 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Enfocing host reboot >2013-02-28 13:53:21,555 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Setup validation >2013-02-28 13:53:21,706 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Hardware supports virtualization >2013-02-28 13:53:21,925 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Failed to execute stage 'Setup validation': Cannot resolve engine host name 'gadi-rhevm.scl.lab.tlv.redhat.com' >2013-02-28 13:53:21,940 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Pre-termination >2013-02-28 13:53:21,969 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20130228135321-10.35.160.152-4c6ece9f.log' >2013-02-28 13:53:22,094 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Termination >2013-02-28 13:53:22,192 ERROR [org.ovirt.engine.core.utils.ssh.SSHDialog] (pool-3-thread-50) SSH error running command root@10.35.160.152:'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && python -c "import sys, tarfile; tarfile.open(fileobj=sys.stdin, mode='r|').extractall(path='${MYTMP}')" && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True': java.io.IOException: Command returned failure code 1 during SSH session 'root@10.35.160.152' > at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:506) [engine-utils.jar:] > at org.ovirt.engine.core.utils.ssh.SSHDialog.executeCommand(SSHDialog.java:308) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsDeploy.execute(VdsDeploy.java:916) [engine-bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:164) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1037) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1122) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1507) [engine-bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166) [engine-utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:108) [engine-utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1140) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:315) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:361) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:566) [engine-bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:209) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:361) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:192) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:181) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:162) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:71) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_09-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2013-02-28 13:53:22,200 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (pool-3-thread-50) [4c6ece9f] Error during host 10.35.160.152 install: java.io.IOException: Command returned failure code 1 during SSH session 'root@10.35.160.152' > at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:506) [engine-utils.jar:] > at org.ovirt.engine.core.utils.ssh.SSHDialog.executeCommand(SSHDialog.java:308) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsDeploy.execute(VdsDeploy.java:916) [engine-bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:164) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1037) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1122) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1507) [engine-bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166) [engine-utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:108) [engine-utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1140) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:315) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:361) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:566) [engine-bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:209) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:361) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:192) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:181) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:162) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:71) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_09-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2013-02-28 13:53:22,215 ERROR [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-50) [4c6ece9f] Host installation failed for host a0688005-fabc-49e6-8d53-e27e40ce06de, cheetah01.scl.lab.tlv.redhat.com.: java.io.IOException: Command returned failure code 1 during SSH session 'root@10.35.160.152' > at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:506) [engine-utils.jar:] > at org.ovirt.engine.core.utils.ssh.SSHDialog.executeCommand(SSHDialog.java:308) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsDeploy.execute(VdsDeploy.java:916) [engine-bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:164) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1037) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1122) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1507) [engine-bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166) [engine-utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:108) [engine-utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1140) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:315) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:361) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:566) [engine-bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:209) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:361) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:192) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:181) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:162) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:71) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_09-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2013-02-28 13:53:29,011 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:29,040 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:39,118 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:39,135 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:49,235 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:49,256 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:53:59,328 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 13:53:59,344 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:54:09,433 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 13:54:09,450 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:54:19,535 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 13:54:19,556 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:54:29,639 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 13:54:29,659 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:54:39,736 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 13:54:39,746 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:54:49,841 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 13:54:49,877 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:54:59,948 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 13:54:59,970 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:55:10,052 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 13:55:10,072 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:55:20,166 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 13:55:20,174 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:55:30,257 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 13:55:30,275 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:55:40,338 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 13:55:40,354 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:55:50,440 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 13:55:50,449 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:56:00,529 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 13:56:00,540 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:56:10,613 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 13:56:10,627 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:56:20,723 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 13:56:20,732 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:56:30,817 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 13:56:30,831 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:56:40,915 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 13:56:40,933 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:56:50,999 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 13:56:51,015 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:57:01,113 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 13:57:01,128 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:57:11,228 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 13:57:11,244 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:57:21,324 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 13:57:21,350 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:57:31,416 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 13:57:31,424 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:57:41,510 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 13:57:41,531 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:57:51,606 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 13:57:51,616 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:58:01,689 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 13:58:01,700 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:58:11,783 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 13:58:11,791 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:58:21,858 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 13:58:21,866 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:58:31,944 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 13:58:31,963 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:58:42,036 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 13:58:42,049 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:58:52,129 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 13:58:52,149 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:59:02,224 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 13:59:02,236 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:59:12,321 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 13:59:12,347 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:59:22,432 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 13:59:22,446 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:59:32,531 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 13:59:32,547 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:59:42,625 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 13:59:42,645 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 13:59:52,720 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 13:59:52,744 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:00:02,828 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:00:02,854 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:00:12,946 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:00:12,966 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:00:23,050 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:00:23,057 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:00:33,139 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:00:33,160 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:00:43,241 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:00:43,260 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:00:53,355 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:00:53,375 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:01:03,447 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:01:03,471 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:01:13,554 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:01:13,568 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:01:23,652 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:01:23,665 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:01:33,757 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:01:33,771 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:01:43,848 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:01:43,865 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:01:53,931 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:01:53,949 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:02:04,028 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:02:04,043 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:02:14,120 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:02:14,128 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:02:24,201 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:02:24,221 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:02:34,308 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:02:34,335 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:02:44,420 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:02:44,429 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:02:54,504 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:02:54,516 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:03:04,601 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:03:04,621 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:03:14,716 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:03:14,734 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:03:24,806 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:03:24,819 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:03:34,907 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:03:34,915 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:03:44,998 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:03:45,019 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:03:55,093 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:03:55,121 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:04:05,205 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:04:05,221 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:04:15,315 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:04:15,333 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:04:25,417 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:04:25,431 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:04:35,508 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:04:35,522 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:04:45,598 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:04:45,611 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:04:55,704 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:04:55,717 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:05:05,806 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:05:05,819 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:05:15,899 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:05:15,910 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:05:25,991 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:05:26,017 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:05:36,096 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:05:36,110 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:05:46,197 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:05:46,218 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:05:56,302 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:05:56,326 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:06:06,405 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:06:06,420 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:06:16,508 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:06:16,527 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:06:26,622 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:06:26,650 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:06:36,725 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:06:36,737 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:06:46,815 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:06:46,828 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:06:56,910 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:06:56,929 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:07:07,009 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:07:07,028 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:07:17,111 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:07:17,124 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:07:27,205 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:07:27,223 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:07:37,307 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:07:37,326 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:07:47,408 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:07:47,434 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:07:57,516 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:07:57,531 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:08:07,623 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:08:07,636 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:08:17,721 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:08:17,734 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:08:27,811 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:08:27,827 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:08:37,911 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:08:37,929 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:08:48,009 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:08:48,018 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:08:58,090 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:08:58,103 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:09:08,181 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:09:08,199 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:09:18,286 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:09:18,299 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:09:28,390 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:09:28,402 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:09:38,493 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:09:38,510 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:09:48,601 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:09:48,613 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:09:58,698 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:09:58,714 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:10:08,793 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:10:08,812 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:27:27,075 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginAdminUserCommand internal: false. >2013-02-28 14:27:27,202 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:27:27,213 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:27:27,214 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:27:27,239 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ajp-/127.0.0.1:8702-7) Failed to decrypt Data must start with zero >2013-02-28 14:27:27,240 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-7) Failed to decrypt value for property LocalAdminPassword will be used encrypted value >2013-02-28 14:27:27,290 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ajp-/127.0.0.1:8702-7) Failed to decrypt Data must start with zero >2013-02-28 14:27:27,290 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-7) Failed to decrypt value for property LocalAdminPassword will be used encrypted value >2013-02-28 14:27:27,349 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ajp-/127.0.0.1:8702-7) Failed to decrypt Data must start with zero >2013-02-28 14:27:27,349 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-7) Failed to decrypt value for property LocalAdminPassword will be used encrypted value >2013-02-28 14:27:27,400 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ajp-/127.0.0.1:8702-7) Failed to decrypt Data must start with zero >2013-02-28 14:27:27,400 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-7) Failed to decrypt value for property LocalAdminPassword will be used encrypted value >2013-02-28 14:27:29,390 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (WebAdminUpperRightButtonURL) with null version, using default general for version >2013-02-28 14:28:29,442 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:28:29,457 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:28:29,458 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:29:29,450 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:29:29,473 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:29:29,474 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:30:29,440 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:30:29,457 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:30:29,458 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:31:29,438 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:31:29,451 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:31:29,452 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:32:29,447 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:32:29,467 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:32:29,468 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:33:29,446 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:33:29,460 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:33:29,461 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:34:29,472 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:34:29,497 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:34:29,498 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:35:29,453 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:35:29,478 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:35:29,478 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:36:08,034 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-4) Lock Acquired to object EngineLock [exclusiveLocks= key: a0688005-fabc-49e6-8d53-e27e40ce06de value: VDS >, sharedLocks= ] >2013-02-28 14:36:08,061 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-3-thread-50) [fafc83d] Running command: RemoveVdsCommand internal: false. Entities affected : ID: a0688005-fabc-49e6-8d53-e27e40ce06de Type: VDS >2013-02-28 14:36:08,154 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-3-thread-50) [fafc83d] START, RemoveVdsVDSCommand( HostId = a0688005-fabc-49e6-8d53-e27e40ce06de), log id: 1702a6d5 >2013-02-28 14:36:08,156 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-3-thread-50) [fafc83d] vdsManager::disposing >2013-02-28 14:36:08,158 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-3-thread-50) [fafc83d] FINISH, RemoveVdsVDSCommand, log id: 1702a6d5 >2013-02-28 14:36:08,161 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-3-thread-50) [fafc83d] Lock freed to object EngineLock [exclusiveLocks= key: a0688005-fabc-49e6-8d53-e27e40ce06de value: VDS >, sharedLocks= ] >2013-02-28 14:36:12,098 INFO [org.ovirt.engine.core.bll.RemoveVdsGroupCommand] (pool-3-thread-50) [5adf7593] Running command: RemoveVdsGroupCommand internal: false. Entities affected : ID: 78382612-3f27-4ea7-960c-f512fecd553c Type: VdsGroups >2013-02-28 14:36:12,435 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:12,454 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:16,085 INFO [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (pool-3-thread-50) [7e4894e2] Running command: RemoveStoragePoolCommand internal: false. Entities affected : ID: adf1b443-62a2-486f-ad51-693efa6441c8 Type: StoragePool >2013-02-28 14:36:16,591 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:16,604 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:16,667 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:16,683 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:16,766 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:16,784 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:16,784 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-02-28 14:36:18,348 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:18,371 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:18,444 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:18,456 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:18,529 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:18,543 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:18,613 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:18,637 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:18,704 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:18,725 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:18,815 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:18,827 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:18,905 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:18,922 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:18,999 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:19,012 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:19,075 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:19,083 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:19,144 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:19,153 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:19,221 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:19,239 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:19,322 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:19,338 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:19,407 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:19,423 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:19,496 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:19,515 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:19,600 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:19,620 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:19,702 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:19,715 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:19,795 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:19,813 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:19,889 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:19,915 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:19,996 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:20,011 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:20,091 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:20,109 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:20,188 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:20,207 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:20,292 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:20,311 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:20,396 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:20,409 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:20,500 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:20,513 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:20,592 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:20,612 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:20,680 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:20,696 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:20,784 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:20,799 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:20,888 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:20,907 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:20,990 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:21,004 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:21,073 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:21,090 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:21,176 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:21,198 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:21,275 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:21,287 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:21,353 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:21,368 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:21,470 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:21,487 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:21,555 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:21,564 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:21,638 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:21,647 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:21,712 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:21,723 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:21,796 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:21,816 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:21,896 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:21,907 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:21,987 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:22,006 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:22,078 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:22,095 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:22,172 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:22,192 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:22,257 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:22,276 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:22,354 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:22,373 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:22,458 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:22,476 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:22,560 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:22,607 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:22,685 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:22,706 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:22,776 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:22,785 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:22,840 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:22,857 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:22,930 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:22,945 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:23,071 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:23,090 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:23,156 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:23,178 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:23,208 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-2) [7a5ce623] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-02-28 14:36:23,293 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:23,311 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:23,407 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:23,424 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:23,505 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:23,519 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:23,604 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:23,622 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:23,677 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:23,695 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:23,721 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-4) [5400209f] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: a84c1822-abfe-44fd-8731-b3a96569303a Type: StoragePool >2013-02-28 14:36:23,834 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:23,848 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:23,925 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:23,947 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:24,022 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:24,034 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:24,719 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:24,736 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:24,806 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:24,821 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:24,969 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-2) [4186d573] Running command: AddVdsCommand internal: false. Entities affected : ID: 3b92277d-98fc-4dda-9680-8c731bcce7ff Type: VdsGroups >2013-02-28 14:36:25,007 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-2) [1af4f8bc] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 50c8e10a-6847-4f03-95a2-ac7e6f4e6fca Type: VDS >2013-02-28 14:36:25,019 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-2) [1af4f8bc] Lock freed to object EngineLock [exclusiveLocks= key: a84c1822-abfe-44fd-8731-b3a96569303a value: REGISTER_VDS >, sharedLocks= ] >2013-02-28 14:36:25,020 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-2) [1af4f8bc] Cannot get vdsManager for vdsid=50c8e10a-6847-4f03-95a2-ac7e6f4e6fca >2013-02-28 14:36:25,022 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [1af4f8bc] START, RemoveVdsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 50c8e10a-6847-4f03-95a2-ac7e6f4e6fca), log id: 4fc83180 >2013-02-28 14:36:25,022 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-2) [1af4f8bc] Cannot get vdsManager for vdsid=50c8e10a-6847-4f03-95a2-ac7e6f4e6fca >2013-02-28 14:36:25,022 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [1af4f8bc] FINISH, RemoveVdsVDSCommand, log id: 4fc83180 >2013-02-28 14:36:25,023 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-2) [1af4f8bc] Cannot get vdsManager for vdsid=50c8e10a-6847-4f03-95a2-ac7e6f4e6fca >2013-02-28 14:36:25,024 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [1af4f8bc] START, AddVdsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 50c8e10a-6847-4f03-95a2-ac7e6f4e6fca), log id: 905bc94 >2013-02-28 14:36:25,024 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [1af4f8bc] AddVds - entered , starting logic to add VDS 50c8e10a-6847-4f03-95a2-ac7e6f4e6fca >2013-02-28 14:36:25,025 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [1af4f8bc] AddVds - VDS 50c8e10a-6847-4f03-95a2-ac7e6f4e6fca was added, will try to add it to the resource manager >2013-02-28 14:36:25,026 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-2) [1af4f8bc] Eneterd VdsManager:constructor >2013-02-28 14:36:25,027 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-2) [1af4f8bc] vdsBroker(10.35.160.152,54,321) >2013-02-28 14:36:25,030 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-2) [1af4f8bc] ResourceManager::AddVds - VDS 50c8e10a-6847-4f03-95a2-ac7e6f4e6fca was added to the Resource Manager >2013-02-28 14:36:25,030 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-2) [1af4f8bc] FINISH, AddVdsVDSCommand, log id: 905bc94 >2013-02-28 14:36:25,057 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-50) [7a92df6f] Running command: InstallVdsCommand internal: true. Entities affected : ID: 50c8e10a-6847-4f03-95a2-ac7e6f4e6fca Type: VDS >2013-02-28 14:36:25,057 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-50) [7a92df6f] Before Installation host 50c8e10a-6847-4f03-95a2-ac7e6f4e6fca, cheetah01.scl.lab.tlv.redhat.com >2013-02-28 14:36:25,098 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:25,118 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:25,149 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-3-thread-50) [7a92df6f] Installation 10.35.160.152: Connected to host 10.35.160.152 with SSH key fingerprint: 13:fc:73:f1:4a:7c:f5:6a:13:3c:b9:e2:50:10:73:7f >2013-02-28 14:36:25,180 INFO [org.ovirt.engine.core.bll.VdsDeploy] (pool-3-thread-50) [7a92df6f] Installation of 10.35.160.152. Executing command via SSH umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && python -c "import sys, tarfile; tarfile.open(fileobj=sys.stdin, mode='r|').extractall(path='${MYTMP}')" && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2013-02-28 14:36:25,190 INFO [org.ovirt.engine.core.utils.ssh.SSHDialog] (pool-3-thread-50) SSH execute root@10.35.160.152 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && python -c "import sys, tarfile; tarfile.open(fileobj=sys.stdin, mode='r|').extractall(path='${MYTMP}')" && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2013-02-28 14:36:25,191 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:25,203 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:25,272 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:25,283 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:25,815 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Initializing >2013-02-28 14:36:25,836 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Environment setup >2013-02-28 14:36:25,847 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Installation packages setup >2013-02-28 14:36:28,035 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-5) Initializing Host: cheetah01.scl.lab.tlv.redhat.com >2013-02-28 14:36:28,396 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Programs detection >2013-02-28 14:36:28,409 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Environment customization >2013-02-28 14:36:28,433 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Logs at host located at: '/tmp/ovirt-host-deploy-20130228143628.log' >2013-02-28 14:36:28,482 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host 10.35.160.152 reports unique id 75f723b4-e40e-4955-b7f1-ab62144e9278 >2013-02-28 14:36:28,502 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 75f723b4-e40e-4955-b7f1-ab62144e9278 to Host 10.35.160.152 >2013-02-28 14:36:28,554 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Enfocing host reboot >2013-02-28 14:36:28,566 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Setup validation >2013-02-28 14:36:28,820 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Hardware supports virtualization >2013-02-28 14:36:28,832 ERROR [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Failed to execute stage 'Setup validation': Cannot resolve engine host name 'gadi-rhevm.scl.lab.tlv.redhat.com' >2013-02-28 14:36:28,843 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Pre-termination >2013-02-28 14:36:28,874 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20130228143628-10.35.160.152-7a92df6f.log' >2013-02-28 14:36:29,007 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Termination >2013-02-28 14:36:29,096 ERROR [org.ovirt.engine.core.utils.ssh.SSHDialog] (pool-3-thread-50) SSH error running command root@10.35.160.152:'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && python -c "import sys, tarfile; tarfile.open(fileobj=sys.stdin, mode='r|').extractall(path='${MYTMP}')" && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True': java.io.IOException: Command returned failure code 1 during SSH session 'root@10.35.160.152' > at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:506) [engine-utils.jar:] > at org.ovirt.engine.core.utils.ssh.SSHDialog.executeCommand(SSHDialog.java:308) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsDeploy.execute(VdsDeploy.java:916) [engine-bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:164) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1037) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1122) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1507) [engine-bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166) [engine-utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:108) [engine-utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1140) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:315) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:361) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:566) [engine-bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:209) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:361) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:192) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:181) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:162) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:71) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_09-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2013-02-28 14:36:29,100 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (pool-3-thread-50) [7a92df6f] Error during host 10.35.160.152 install: java.io.IOException: Command returned failure code 1 during SSH session 'root@10.35.160.152' > at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:506) [engine-utils.jar:] > at org.ovirt.engine.core.utils.ssh.SSHDialog.executeCommand(SSHDialog.java:308) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsDeploy.execute(VdsDeploy.java:916) [engine-bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:164) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1037) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1122) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1507) [engine-bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166) [engine-utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:108) [engine-utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1140) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:315) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:361) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:566) [engine-bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:209) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:361) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:192) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:181) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:162) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:71) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_09-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2013-02-28 14:36:29,106 ERROR [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-50) [7a92df6f] Host installation failed for host 50c8e10a-6847-4f03-95a2-ac7e6f4e6fca, cheetah01.scl.lab.tlv.redhat.com.: java.io.IOException: Command returned failure code 1 during SSH session 'root@10.35.160.152' > at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:506) [engine-utils.jar:] > at org.ovirt.engine.core.utils.ssh.SSHDialog.executeCommand(SSHDialog.java:308) [engine-utils.jar:] > at org.ovirt.engine.core.bll.VdsDeploy.execute(VdsDeploy.java:916) [engine-bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:164) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1037) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1122) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1507) [engine-bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:166) [engine-utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:108) [engine-utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1140) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:315) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:361) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:566) [engine-bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:209) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:361) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:192) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:181) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand$3.run(AddVdsCommand.java:162) [engine-bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:71) [engine-utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_09-icedtea] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_09-icedtea] > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2013-02-28 14:36:29,456 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:29,471 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:29,472 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:36:35,367 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:35,385 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:45,464 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:45,478 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:36:55,570 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:36:55,582 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:37:03,165 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-33) Start refreshing all users data >2013-02-28 14:37:03,238 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2013-02-28 14:37:05,669 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:37:05,685 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:37:15,775 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:37:15,788 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:37:25,874 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:37:25,883 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:37:29,457 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:37:29,463 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:37:29,464 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:37:35,966 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:37:35,984 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:37:46,066 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:37:46,086 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:37:56,171 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:37:56,183 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:38:06,268 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:38:06,283 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:38:16,360 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:38:16,378 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:38:26,453 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:38:26,470 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:38:29,461 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:38:29,484 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:38:29,484 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:38:36,553 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:38:36,564 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:38:46,639 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:38:46,656 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:38:56,747 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:38:56,777 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:39:06,865 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:39:06,883 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:39:16,966 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:39:16,987 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:39:27,061 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:39:27,077 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:39:29,463 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:39:29,478 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:39:29,479 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:39:37,166 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:39:37,181 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:39:47,252 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:39:47,261 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:39:57,336 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:39:57,351 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:40:07,435 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:40:07,448 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:40:17,535 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:40:17,552 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:40:27,627 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:40:27,642 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:40:29,466 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:40:29,478 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:40:29,479 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:40:37,736 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:40:37,752 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:40:47,836 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:40:47,848 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:40:57,935 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:40:57,948 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:41:08,020 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:41:08,036 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:41:18,111 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:41:18,128 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:41:28,217 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:41:28,228 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:41:29,469 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:41:29,481 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:41:29,481 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:41:38,311 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:41:38,327 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:41:48,404 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:41:48,412 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:41:58,477 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:41:58,494 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:42:08,581 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:42:08,598 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:42:18,686 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:42:18,701 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:42:28,789 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:42:28,812 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:42:29,472 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:42:29,484 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:42:29,485 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:42:38,895 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:42:38,905 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:42:48,966 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:42:48,978 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:42:59,052 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:42:59,068 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:43:09,153 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:43:09,164 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:43:19,259 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:43:19,282 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:43:29,364 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:43:29,381 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:43:29,473 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:43:29,485 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:43:29,486 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:43:39,469 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:43:39,485 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:43:49,574 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:43:49,591 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:43:59,670 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:43:59,681 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:44:09,765 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:44:09,786 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:44:19,858 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:44:19,874 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:44:29,475 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:44:29,491 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:44:29,492 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:44:29,960 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:44:29,972 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:44:40,059 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:44:40,071 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:44:50,155 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:44:50,171 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:45:00,263 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:45:00,277 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:45:10,359 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:45:10,371 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:45:20,441 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:45:20,458 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:45:29,478 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:45:29,495 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:45:29,496 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:45:30,527 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:45:30,538 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:45:40,625 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:45:40,646 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:45:50,736 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:45:50,756 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:46:00,845 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:46:00,861 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:46:10,942 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:46:10,960 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:46:21,039 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:46:21,052 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:46:29,480 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:46:29,499 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:46:29,500 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:46:31,136 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:46:31,146 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:46:41,232 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:46:41,244 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:46:51,327 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:46:51,340 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:47:01,414 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:47:01,425 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:47:11,514 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:47:11,521 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:47:21,590 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:47:21,607 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:47:29,483 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:47:29,495 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:47:29,495 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:47:31,687 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:47:31,699 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:47:41,784 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:47:41,792 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:47:51,870 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:47:51,885 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:48:01,972 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:48:01,986 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:48:12,058 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-02-28 14:48:12,075 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:48:22,162 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:48:22,180 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:48:29,484 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:48:29,501 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:48:29,502 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:48:32,265 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:48:32,276 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:48:42,354 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:48:42,374 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:48:52,446 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:48:52,470 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:49:02,540 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:49:02,560 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:49:12,645 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:49:12,661 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:49:22,746 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:49:22,760 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:49:29,486 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:49:29,508 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:49:29,509 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:49:32,844 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:49:32,861 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:49:42,939 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:49:42,950 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:49:53,025 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-02-28 14:49:53,044 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:50:03,125 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:50:03,136 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:50:13,220 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:50:13,231 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:50:23,315 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:50:23,328 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:50:29,488 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:50:29,501 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:50:29,502 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:50:33,417 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-02-28 14:50:33,430 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:50:43,513 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:50:43,529 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:50:53,618 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-02-28 14:50:53,634 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:51:03,725 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-02-28 14:51:03,741 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:51:13,835 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:51:13,859 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:51:23,936 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:51:23,948 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:51:29,490 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:51:29,498 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:51:29,499 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-02-28 14:51:34,030 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-02-28 14:51:34,047 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:51:44,128 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-02-28 14:51:44,144 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-02-28 14:51:54,233 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-02-28 14:51:54,248 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 11:16:00,400 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 34) Start time: 3/5/13 11:16 AM >2013-03-05 11:16:01,384 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ServerService Thread Pool -- 34) Failed to decrypt Data must start with zero >2013-03-05 11:16:01,391 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ServerService Thread Pool -- 34) Failed to decrypt value for property TruststorePass will be used encrypted value >2013-03-05 11:16:01,528 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 34) VDSBrokerFrontend: 3/5/13 11:16 AM >2013-03-05 11:16:01,541 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 34) CpuFlagsManager: 3/5/13 11:16 AM >2013-03-05 11:16:01,543 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 34) Setting audit clean up manager to run at: 35 35 3 * * ? >2013-03-05 11:16:01,604 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 34) AuditLogCleanupManager: 3/5/13 11:16 AM >2013-03-05 11:16:01,609 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 34) TagsDirector initialization >2013-03-05 11:16:01,674 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 34) Tag root added to tree >2013-03-05 11:16:01,702 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 34) TagsDirector: 3/5/13 11:16 AM >2013-03-05 11:16:01,723 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 34) IsoDomainListSyncronizer: 3/5/13 11:16 AM >2013-03-05 11:16:01,789 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 34) InitHandlers: 3/5/13 11:16 AM >2013-03-05 11:16:01,826 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 34) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2013-03-05 11:16:01,829 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 34) ErrorTranslator: 3/5/13 11:16 AM >2013-03-05 11:16:01,829 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 34) VdsErrorTranslator: 3/5/13 11:16 AM >2013-03-05 11:16:01,830 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 34) Mark uncompleted jobs as UNKNOWN: 3/5/13 11:16 AM >2013-03-05 11:16:01,910 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 34) JobRepositoryCleanupManager: 3/5/13 11:16 AM >2013-03-05 11:16:01,920 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 34) AutoRecoveryManager: 3/5/13 11:16 AM >2013-03-05 11:16:01,932 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 34) ExecutionMessageDirector: 3/5/13 11:16 AM >2013-03-05 11:16:01,996 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (ServerService Thread Pool -- 32) UsersDomainsCacheManager: 3/5/13 11:16 AM >2013-03-05 11:16:02,008 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 35) InitResourceManager: 3/5/13 11:16 AM >2013-03-05 11:16:02,020 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 35) ResourceManager::ResourceManager::Entered >2013-03-05 11:16:02,117 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (ServerService Thread Pool -- 32) DbUserCacheManager: 3/5/13 11:16 AM >2013-03-05 11:16:02,165 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-1) Start refreshing all users data >2013-03-05 11:16:02,343 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ServerService Thread Pool -- 35) AsyncTaskManager: Initialization of AsyncTaskManager completed successfully. >2013-03-05 11:16:02,345 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 35) AsyncTaskManager: 3/5/13 11:16 AM >2013-03-05 11:16:02,359 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (ServerService Thread Pool -- 35) Initialization of OvfDataUpdater completed successfully. >2013-03-05 11:16:02,360 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 35) OvfDataUpdater: 3/5/13 11:16 AM >2013-03-05 11:16:02,369 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 35) VdsLoadBalancer: 3/5/13 11:16 AM >2013-03-05 11:16:02,378 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (pool-3-thread-1) MacPoolManager started: 3/5/13 11:16 AM >2013-03-05 11:16:02,432 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 35) Init VM Custom Properties utilities: 3/5/13 11:16 AM >2013-03-05 11:16:02,479 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (pool-3-thread-1) MacPoolManager finished: 3/5/13 11:16 AM >2013-03-05 12:16:02,197 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-66) Start refreshing all users data >2013-03-05 12:16:02,359 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2013-03-05 12:35:43,692 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Loaded file "/usr/share/ovirt-engine/conf/engine.conf.defaults". >2013-03-05 12:35:43,693 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Loaded file "/etc/sysconfig/ovirt-engine". >2013-03-05 12:35:43,693 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine". >2013-03-05 12:35:43,694 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_PROPERTIES" is "". >2013-03-05 12:35:43,694 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2013-03-05 12:35:43,694 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_PERM_MIN" is "256m". >2013-03-05 12:35:43,694 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "JBOSS_HOME" is "/usr/share/jbossas". >2013-03-05 12:35:43,694 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_DB_USER" is "engine". >2013-03-05 12:35:43,694 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2013-03-05 12:35:43,694 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2013-03-05 12:35:43,695 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_STOP_TIME" is "10". >2013-03-05 12:35:43,695 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_FQDN" is "gadi-rhevm.scl.lab.tlv.redhat.com". >2013-03-05 12:35:43,695 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_HEAP_MAX" is "1g". >2013-03-05 12:35:43,695 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2013-03-05 12:35:43,695 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_LOCK" is "/var/lock/ovirt-engine". >2013-03-05 12:35:43,695 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_STOP_INTERVAL" is "1". >2013-03-05 12:35:43,695 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_PROXY_ENABLED" is "true". >2013-03-05 12:35:43,695 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2013-03-05 12:35:43,696 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_APPS" is "engine.ear". >2013-03-05 12:35:43,696 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_DB_PASSWORD" is "64c5fd2979a86168". >2013-03-05 12:35:43,696 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2013-03-05 12:35:43,696 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2013-03-05 12:35:43,696 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2013-03-05 12:35:43,696 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_HTTP_PORT" is "8700". >2013-03-05 12:35:43,702 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_AJP_PORT" is "8702". >2013-03-05 12:35:43,703 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_PERM_MAX" is "256m". >2013-03-05 12:35:43,703 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_VERBOSE_GC" is "false". >2013-03-05 12:35:43,703 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_HTTPS_PORT" is "8701". >2013-03-05 12:35:43,703 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2013-03-05 12:35:43,703 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2013-03-05 12:35:43,703 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_NOFILE" is "65535". >2013-03-05 12:35:43,703 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2013-03-05 12:35:43,703 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2013-03-05 12:35:43,704 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_PID" is "/var/run/ovirt-engine.pid". >2013-03-05 12:35:43,704 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2013-03-05 12:35:43,704 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_HTTP_ENABLED" is "false". >2013-03-05 12:35:43,704 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_GROUP" is "ovirt". >2013-03-05 12:35:43,704 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "JAVA_HOME" is "/usr/lib/jvm/jre-1.7.0-openjdk.x86_64". >2013-03-05 12:35:43,704 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2013-03-05 12:35:43,704 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_DB_SSL" is "false". >2013-03-05 12:35:43,704 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2013-03-05 12:35:43,705 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_HEAP_MIN" is "1g". >2013-03-05 12:35:43,705 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_AJP_ENABLED" is "true". >2013-03-05 12:35:43,705 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-4) Value of property "ENGINE_USER" is "ovirt". >2013-03-05 12:35:50,589 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginAdminUserCommand internal: false. >2013-03-05 12:35:50,679 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: UNASSIGNED not exist in string table >2013-03-05 12:35:50,680 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2013-03-05 12:35:50,681 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2013-03-05 12:35:50,681 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2013-03-05 12:35:50,681 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: DWH_STOPPED not exist in string table >2013-03-05 12:35:50,681 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: DWH_STARTED not exist in string table >2013-03-05 12:35:50,681 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: DWH_ERROR not exist in string table >2013-03-05 12:35:50,681 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: EXTERNAL_EVENT_NORMAL not exist in string table >2013-03-05 12:35:50,682 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: EXTERNAL_EVENT_WARNING not exist in string table >2013-03-05 12:35:50,682 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: EXTERNAL_EVENT_ERROR not exist in string table >2013-03-05 12:35:50,682 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: EXTERNAL_ALERT not exist in string table >2013-03-05 12:35:50,692 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: USER_ATTACH_DISK_TO_VM not have severity. Assumed Normal >2013-03-05 12:35:50,692 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: USER_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2013-03-05 12:35:50,693 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: USER_FAILED_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2013-03-05 12:35:50,693 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not have severity. Assumed Normal >2013-03-05 12:35:50,693 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: VM_MIGRATION_START_SYSTEM_INITIATED not have severity. Assumed Normal >2013-03-05 12:35:50,693 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not have severity. Assumed Normal >2013-03-05 12:35:50,694 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: NETWORK_UPDATE_VM_INTERFACE_LINK_UP not have severity. Assumed Normal >2013-03-05 12:35:50,694 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) AuditLogType: NETWORK_UPDATE_VM_INTERFACE_LINK_DOWN not have severity. Assumed Normal >2013-03-05 12:35:50,957 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ajp-/127.0.0.1:8702-3) Failed to decrypt Data must start with zero >2013-03-05 12:35:50,964 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-3) Failed to decrypt value for property LocalAdminPassword will be used encrypted value >2013-03-05 12:35:51,076 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ajp-/127.0.0.1:8702-3) Failed to decrypt Data must start with zero >2013-03-05 12:35:51,076 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-3) Failed to decrypt value for property LocalAdminPassword will be used encrypted value >2013-03-05 12:35:51,200 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ajp-/127.0.0.1:8702-3) Failed to decrypt Data must start with zero >2013-03-05 12:35:51,201 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-3) Failed to decrypt value for property LocalAdminPassword will be used encrypted value >2013-03-05 12:35:51,294 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ajp-/127.0.0.1:8702-3) Failed to decrypt Data must start with zero >2013-03-05 12:35:51,295 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-3) Failed to decrypt value for property LocalAdminPassword will be used encrypted value >2013-03-05 12:35:51,795 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-05 12:35:51,816 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:35:51,841 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:35:53,299 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (WebAdminUpperRightButtonURL) with null version, using default general for version >2013-03-05 12:36:54,182 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-05 12:36:54,226 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:36:54,228 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:37:54,173 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-05 12:37:54,199 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:37:54,205 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:38:54,173 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 12:38:54,229 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:38:54,230 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:39:54,174 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 12:39:54,212 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:39:54,214 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:40:54,179 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 12:40:54,197 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:40:54,198 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:41:45,014 INFO [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp-/127.0.0.1:8702-3) [71666e6a] Running command: UpdateVdsGroupCommand internal: false. Entities affected : ID: 99408929-82cf-4dc7-a532-9d998063fa95 Type: VdsGroups >2013-03-05 12:41:54,190 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 12:41:54,225 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:41:54,227 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:42:04,075 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-3) [313d7a1a] Running command: AddVdsCommand internal: false. Entities affected : ID: 99408929-82cf-4dc7-a532-9d998063fa95 Type: VdsGroups >2013-03-05 12:42:04,250 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-3) [5b49479e] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-05 12:42:04,272 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-3) [5b49479e] Lock freed to object EngineLock [exclusiveLocks= key: 5849b030-626e-47cb-ad90-3ce782d831b3 value: REGISTER_VDS >, sharedLocks= ] >2013-03-05 12:42:04,279 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-3) [5b49479e] Cannot get vdsManager for vdsid=cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 >2013-03-05 12:42:04,294 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-3) [5b49479e] START, RemoveVdsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0), log id: 32aca6d3 >2013-03-05 12:42:04,295 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-3) [5b49479e] Cannot get vdsManager for vdsid=cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 >2013-03-05 12:42:04,295 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-3) [5b49479e] FINISH, RemoveVdsVDSCommand, log id: 32aca6d3 >2013-03-05 12:42:04,296 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-3) [5b49479e] Cannot get vdsManager for vdsid=cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 >2013-03-05 12:42:04,303 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-3) [5b49479e] START, AddVdsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0), log id: 261cfc15 >2013-03-05 12:42:04,303 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-3) [5b49479e] AddVds - entered , starting logic to add VDS cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 >2013-03-05 12:42:04,306 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-3) [5b49479e] AddVds - VDS cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 was added, will try to add it to the resource manager >2013-03-05 12:42:04,313 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-3) [5b49479e] Eneterd VdsManager:constructor >2013-03-05 12:42:04,314 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-3) [5b49479e] vdsBroker(cheetah01.scl.lab.tlv.redhat.com,54,321) >2013-03-05 12:42:04,443 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-3) [5b49479e] ResourceManager::AddVds - VDS cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 was added to the Resource Manager >2013-03-05 12:42:04,448 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-3) [5b49479e] FINISH, AddVdsVDSCommand, log id: 261cfc15 >2013-03-05 12:42:04,549 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-10) [1973499] Running command: InstallVdsCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-05 12:42:04,555 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-10) [1973499] Before Installation host cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, cheetah01.scl.lab.tlv.redhat.com >2013-03-05 12:42:04,697 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-3-thread-10) [1973499] Installation cheetah01.scl.lab.tlv.redhat.com: Connected to host cheetah01.scl.lab.tlv.redhat.com with SSH key fingerprint: 13:fc:73:f1:4a:7c:f5:6a:13:3c:b9:e2:50:10:73:7f >2013-03-05 12:42:04,731 INFO [org.ovirt.engine.core.bll.VdsDeploy] (pool-3-thread-10) [1973499] Installation of cheetah01.scl.lab.tlv.redhat.com. Executing command via SSH umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2013-03-05 12:42:04,732 INFO [org.ovirt.engine.core.utils.archivers.tar.CachedTar] (pool-3-thread-10) Tarball '/var/cache/ovirt-engine/ovirt-host-deploy.tar' is missing, creating >2013-03-05 12:42:04,809 INFO [org.ovirt.engine.core.utils.ssh.SSHDialog] (pool-3-thread-10) SSH execute root@cheetah01.scl.lab.tlv.redhat.com 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2013-03-05 12:42:05,418 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Stage: Initializing >2013-03-05 12:42:05,475 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Stage: Environment setup >2013-03-05 12:42:05,492 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Stage: Installation packages setup >2013-03-05 12:42:07,451 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-97) Initializing Host: cheetah01.scl.lab.tlv.redhat.com >2013-03-05 12:42:08,192 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Stage: Programs detection >2013-03-05 12:42:08,230 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Stage: Environment customization >2013-03-05 12:42:08,254 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Logs at host located at: '/tmp/ovirt-host-deploy-20130305124207.log' >2013-03-05 12:42:08,308 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host cheetah01.scl.lab.tlv.redhat.com reports unique id 75f723b4-e40e-4955-b7f1-ab62144e9278 >2013-03-05 12:42:08,332 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 75f723b4-e40e-4955-b7f1-ab62144e9278 to Host cheetah01.scl.lab.tlv.redhat.com >2013-03-05 12:42:08,392 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Enfocing host reboot >2013-03-05 12:42:08,406 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Stage: Setup validation >2013-03-05 12:42:08,991 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Hardware supports virtualization >2013-03-05 12:42:09,114 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Stage: Transaction setup >2013-03-05 12:42:09,145 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Stage: Package installation >2013-03-05 12:42:09,475 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Yum Status: Downloading Packages >2013-03-05 12:42:09,500 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Yum Download/Verify: qemu-kvm-rhev-tools-0.12.1.2-2.348.el6.x86_64 >2013-03-05 12:42:09,526 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Yum Download/Verify: tuned-0.2.19-11.el6.noarch >2013-03-05 12:42:09,536 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Yum Status: Check Package Signatures >2013-03-05 12:42:09,554 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Yum Status: Running Test Transaction >2013-03-05 12:42:09,615 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Yum Status: Running Transaction >2013-03-05 12:42:09,841 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Yum Installing: 1/2: tuned-0.2.19-11.el6.noarch >2013-03-05 12:42:10,366 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Yum Installing: 2/2: 2:qemu-kvm-rhev-tools-0.12.1.2-2.348.el6.x86_64 >2013-03-05 12:42:11,369 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Yum Verify: 1/2: qemu-kvm-rhev-tools.x86_64 2:0.12.1.2-2.348.el6 - u >2013-03-05 12:42:11,449 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Yum Verify: 2/2: tuned.noarch 0:0.2.19-11.el6 - u >2013-03-05 12:42:11,704 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Stage: Misc configuration >2013-03-05 12:42:11,782 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Setting time >2013-03-05 12:42:27,437 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Enrolling certificate >2013-03-05 12:42:28,824 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Stage: Transaction commit >2013-03-05 12:42:28,837 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Stage: Closing up >2013-03-05 12:42:29,312 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Reboot scheduled >2013-03-05 12:42:29,323 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Stage: Pre-termination >2013-03-05 12:42:29,339 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Reboot scheduled >2013-03-05 12:42:29,354 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20130305124229-cheetah01.scl.lab.tlv.redhat.com-1973499.log' >2013-03-05 12:42:29,732 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah01.scl.lab.tlv.redhat.com: Stage: Termination >2013-03-05 12:42:29,899 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-10) [1973499] After Installation host cheetah01.scl.lab.tlv.redhat.com, VDS >2013-03-05 12:42:29,900 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-11) [1973499] Waiting 300 seconds, for server to finish reboot process. >2013-03-05 12:42:31,779 WARN [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-2) [1389873e] CanDoAction of action AddVds failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__HOST,ACTION_TYPE_FAILED_VDS_WITH_SAME_HOST_EXIST >2013-03-05 12:42:40,005 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-6) [7a9dda18] Running command: AddVdsCommand internal: false. Entities affected : ID: 99408929-82cf-4dc7-a532-9d998063fa95 Type: VdsGroups >2013-03-05 12:42:40,056 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-6) [7de6cbaa] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-05 12:42:40,072 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-6) [7de6cbaa] Lock freed to object EngineLock [exclusiveLocks= key: 5849b030-626e-47cb-ad90-3ce782d831b3 value: REGISTER_VDS >, sharedLocks= ] >2013-03-05 12:42:40,072 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-6) [7de6cbaa] Cannot get vdsManager for vdsid=5e9ad544-fe9f-4669-8b11-1631ba7d1c12 >2013-03-05 12:42:40,075 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [7de6cbaa] START, RemoveVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3b032e8a >2013-03-05 12:42:40,075 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-6) [7de6cbaa] Cannot get vdsManager for vdsid=5e9ad544-fe9f-4669-8b11-1631ba7d1c12 >2013-03-05 12:42:40,075 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [7de6cbaa] FINISH, RemoveVdsVDSCommand, log id: 3b032e8a >2013-03-05 12:42:40,076 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-6) [7de6cbaa] Cannot get vdsManager for vdsid=5e9ad544-fe9f-4669-8b11-1631ba7d1c12 >2013-03-05 12:42:40,078 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [7de6cbaa] START, AddVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: c45bb02 >2013-03-05 12:42:40,078 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [7de6cbaa] AddVds - entered , starting logic to add VDS 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 >2013-03-05 12:42:40,085 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [7de6cbaa] AddVds - VDS 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 was added, will try to add it to the resource manager >2013-03-05 12:42:40,086 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-6) [7de6cbaa] Eneterd VdsManager:constructor >2013-03-05 12:42:40,086 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-6) [7de6cbaa] vdsBroker(cheetah02.scl.lab.tlv.redhat.com,54,321) >2013-03-05 12:42:40,090 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-6) [7de6cbaa] ResourceManager::AddVds - VDS 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 was added to the Resource Manager >2013-03-05 12:42:40,090 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [7de6cbaa] FINISH, AddVdsVDSCommand, log id: c45bb02 >2013-03-05 12:42:40,148 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-12) [2e50e0f9] Running command: InstallVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-05 12:42:40,151 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-12) [2e50e0f9] Before Installation host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, cheetah02.scl.lab.tlv.redhat.com >2013-03-05 12:42:40,247 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-3-thread-12) [2e50e0f9] Installation cheetah02.scl.lab.tlv.redhat.com: Connected to host cheetah02.scl.lab.tlv.redhat.com with SSH key fingerprint: 5b:9a:58:d2:ba:f3:ea:da:f7:77:52:d7:fa:50:6a:68 >2013-03-05 12:42:40,287 INFO [org.ovirt.engine.core.bll.VdsDeploy] (pool-3-thread-12) [2e50e0f9] Installation of cheetah02.scl.lab.tlv.redhat.com. Executing command via SSH umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2013-03-05 12:42:40,294 INFO [org.ovirt.engine.core.utils.ssh.SSHDialog] (pool-3-thread-12) SSH execute root@cheetah02.scl.lab.tlv.redhat.com 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2013-03-05 12:42:40,922 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Stage: Initializing >2013-03-05 12:42:40,947 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Stage: Environment setup >2013-03-05 12:42:40,965 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Stage: Installation packages setup >2013-03-05 12:42:43,096 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-24) Initializing Host: cheetah02.scl.lab.tlv.redhat.com >2013-03-05 12:42:44,138 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Stage: Programs detection >2013-03-05 12:42:44,154 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Stage: Environment customization >2013-03-05 12:42:44,187 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Logs at host located at: '/tmp/ovirt-host-deploy-20130305124243.log' >2013-03-05 12:42:44,239 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host cheetah02.scl.lab.tlv.redhat.com reports unique id 1bc49e96-c418-4395-88a6-c182350146a2 >2013-03-05 12:42:44,258 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 1bc49e96-c418-4395-88a6-c182350146a2 to Host cheetah02.scl.lab.tlv.redhat.com >2013-03-05 12:42:44,305 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Enfocing host reboot >2013-03-05 12:42:44,318 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Stage: Setup validation >2013-03-05 12:42:44,529 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Hardware supports virtualization >2013-03-05 12:42:44,548 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Stage: Transaction setup >2013-03-05 12:42:44,567 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Stage: Package installation >2013-03-05 12:42:44,911 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Yum Status: Downloading Packages >2013-03-05 12:42:44,927 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Yum Download/Verify: qemu-kvm-rhev-tools-0.12.1.2-2.348.el6.x86_64 >2013-03-05 12:42:44,943 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Yum Download/Verify: tuned-0.2.19-11.el6.noarch >2013-03-05 12:42:44,955 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Yum Status: Check Package Signatures >2013-03-05 12:42:44,961 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Yum Status: Running Test Transaction >2013-03-05 12:42:45,024 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Yum Status: Running Transaction >2013-03-05 12:42:45,240 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Yum Installing: 1/2: tuned-0.2.19-11.el6.noarch >2013-03-05 12:42:45,764 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Yum Installing: 2/2: 2:qemu-kvm-rhev-tools-0.12.1.2-2.348.el6.x86_64 >2013-03-05 12:42:46,732 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Yum Verify: 1/2: qemu-kvm-rhev-tools.x86_64 2:0.12.1.2-2.348.el6 - u >2013-03-05 12:42:46,800 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Yum Verify: 2/2: tuned.noarch 0:0.2.19-11.el6 - u >2013-03-05 12:42:47,022 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Stage: Misc configuration >2013-03-05 12:42:47,058 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Setting time >2013-03-05 12:42:54,170 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-05 12:42:54,181 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:42:54,183 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:43:01,829 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Enrolling certificate >2013-03-05 12:43:03,170 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Stage: Transaction commit >2013-03-05 12:43:03,185 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Stage: Closing up >2013-03-05 12:43:03,581 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Reboot scheduled >2013-03-05 12:43:03,600 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Stage: Pre-termination >2013-03-05 12:43:03,624 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Reboot scheduled >2013-03-05 12:43:03,637 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20130305124303-cheetah02.scl.lab.tlv.redhat.com-2e50e0f9.log' >2013-03-05 12:43:03,782 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation cheetah02.scl.lab.tlv.redhat.com: Stage: Termination >2013-03-05 12:43:03,950 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-12) [2e50e0f9] After Installation host cheetah02.scl.lab.tlv.redhat.com, VDS >2013-03-05 12:43:03,953 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-14) [2e50e0f9] Waiting 300 seconds, for server to finish reboot process. >2013-03-05 12:43:54,217 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-05 12:43:54,235 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:43:54,236 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:44:54,185 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-05 12:44:54,207 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:44:54,209 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:45:54,183 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 12:45:54,214 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:45:54,216 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:46:54,188 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 12:46:54,213 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:46:54,215 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:47:29,913 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-11) [1973499] START, SetVdsStatusVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, status=NonResponsive, nonOperationalReason=NONE), log id: 2bb8cb47 >2013-03-05 12:47:29,937 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-11) [1973499] FINISH, SetVdsStatusVDSCommand, log id: 2bb8cb47 >2013-03-05 12:47:54,190 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-05 12:47:54,220 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:47:54,221 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:48:03,961 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-14) [2e50e0f9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonResponsive, nonOperationalReason=NONE), log id: 26e031eb >2013-03-05 12:48:03,974 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-14) [2e50e0f9] FINISH, SetVdsStatusVDSCommand, log id: 26e031eb >2013-03-05 12:48:54,199 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-05 12:48:54,217 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:48:54,218 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:49:54,195 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-05 12:49:54,223 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:49:54,224 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:50:54,233 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 12:50:54,253 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:50:54,255 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:51:54,189 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-05 12:51:54,201 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:51:54,202 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:52:54,187 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-05 12:52:54,197 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:52:54,198 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:53:54,196 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-05 12:53:54,214 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:53:54,215 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:54:54,194 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 12:54:54,206 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:54:54,207 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:55:54,197 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-05 12:55:54,217 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:55:54,219 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:56:54,199 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-05 12:56:54,222 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:56:54,224 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:57:54,198 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 12:57:54,217 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:57:54,219 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:58:54,202 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 12:58:54,225 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:58:54,227 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 12:59:54,204 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-05 12:59:54,227 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 12:59:54,227 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:00:54,216 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-05 13:00:54,234 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:00:54,240 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:01:54,209 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-05 13:01:54,230 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:01:54,232 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:02:54,206 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-05 13:02:54,224 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:02:54,226 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:03:54,208 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-05 13:03:54,232 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:03:54,232 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:04:54,214 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 13:04:54,232 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:04:54,234 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:05:54,208 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-05 13:05:54,220 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:05:54,220 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:06:54,214 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-05 13:06:54,234 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:06:54,236 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:07:54,216 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 13:07:54,230 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:07:54,230 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:08:54,217 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-05 13:08:54,239 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:08:54,241 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:09:54,219 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-05 13:09:54,239 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:09:54,243 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:10:54,221 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-05 13:10:54,249 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:10:54,250 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:11:54,228 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 13:11:54,246 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:11:54,248 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:12:54,219 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-05 13:12:54,229 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:12:54,230 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:13:54,229 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-05 13:13:54,244 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:13:54,245 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:14:54,227 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-05 13:14:54,242 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:14:54,243 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:15:54,230 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 13:15:54,249 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:15:54,251 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:16:02,208 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-72) Start refreshing all users data >2013-03-05 13:16:02,379 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2013-03-05 13:16:54,227 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-05 13:16:54,240 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:16:54,242 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:17:54,230 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 13:17:54,248 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:17:54,249 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:18:54,231 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 13:18:54,244 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:18:54,245 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:19:54,236 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-05 13:19:54,254 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:19:54,255 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:20:54,236 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-05 13:20:54,251 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:20:54,252 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:21:54,239 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 13:21:54,258 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:21:54,259 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:22:54,242 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-05 13:22:54,269 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:22:54,270 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:23:54,243 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-05 13:23:54,261 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:23:54,263 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:24:54,243 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-05 13:24:54,260 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:24:54,263 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:25:54,250 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 13:25:54,268 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:25:54,269 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:26:40,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: 6d4763dc >2013-03-05 13:26:40,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) FINISH, GetHardwareInfoVDSCommand, log id: 6d4763dc >2013-03-05 13:26:40,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: 5c53d2cb >2013-03-05 13:26:40,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) FINISH, GetHardwareInfoVDSCommand, log id: 5c53d2cb >2013-03-05 13:26:40,992 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-94) [65093d2b] Running command: InitVdsOnUpCommand internal: true. >2013-03-05 13:26:41,037 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-49) [65093d2b] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2013-03-05 13:26:41,045 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [3f31045f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-05 13:26:41,063 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [52ef70c8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-05 13:26:54,242 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 13:26:54,322 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:26:54,323 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:27:54,248 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 13:27:54,273 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:27:54,275 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:28:54,244 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-05 13:28:54,267 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:28:54,267 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:29:54,249 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-05 13:29:54,276 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:29:54,277 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:30:54,247 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 13:30:54,263 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:30:54,265 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:31:54,254 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-05 13:31:54,274 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:31:54,275 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:32:54,241 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 13:32:54,250 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:32:54,250 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:33:54,258 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-05 13:33:54,273 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:33:54,274 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:34:54,263 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-05 13:34:54,275 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:34:54,277 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:35:54,267 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-05 13:35:54,283 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:35:54,285 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:36:54,262 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 13:36:54,278 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:36:54,279 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:37:54,252 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 13:37:54,269 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:37:54,269 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:38:54,258 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 13:38:54,270 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:38:54,270 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:39:54,262 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 13:39:54,277 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:39:54,277 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:40:54,268 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 13:40:54,287 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:40:54,288 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:41:54,267 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-05 13:41:54,287 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:41:54,288 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:42:54,270 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-05 13:42:54,288 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:42:54,290 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:43:54,266 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-05 13:43:54,276 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:43:54,277 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:44:54,272 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-05 13:44:54,288 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:44:54,289 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:45:54,268 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 13:45:54,279 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:45:54,280 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:46:54,277 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-05 13:46:54,292 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:46:54,294 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:47:54,275 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-05 13:47:54,294 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:47:54,295 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:48:54,277 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 13:48:54,291 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:48:54,292 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:49:54,278 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 13:49:54,296 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:49:54,298 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:50:54,275 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-05 13:50:54,283 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:50:54,284 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:51:54,282 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 13:51:54,296 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:51:54,297 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:52:54,284 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-05 13:52:54,297 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:52:54,298 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:53:52,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7c458eb9 >2013-03-05 13:53:52,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) FINISH, GetHardwareInfoVDSCommand, log id: 7c458eb9 >2013-03-05 13:53:52,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6f6b0206 >2013-03-05 13:53:52,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) FINISH, GetHardwareInfoVDSCommand, log id: 6f6b0206 >2013-03-05 13:53:53,091 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-20) [6d44392] Running command: InitVdsOnUpCommand internal: true. >2013-03-05 13:53:53,121 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-50) [6d44392] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2013-03-05 13:53:53,135 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-20) [6d2cc07b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-05 13:53:53,145 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-20) [4bbfe1d7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-05 13:53:54,305 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 13:53:54,321 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:53:54,321 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:54:16,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-7) [41f9b00c] START, ValidateStorageServerConnectionVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1188d5a1 >2013-03-05 13:54:16,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-7) [41f9b00c] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 1188d5a1 >2013-03-05 13:54:16,380 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-7) [41f9b00c] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-05 13:54:16,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [41f9b00c] START, ConnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7d872cb9 >2013-03-05 13:54:16,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [41f9b00c] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=100}, log id: 7d872cb9 >2013-03-05 13:54:16,778 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (ajp-/127.0.0.1:8702-7) [41f9b00c] The connection with details lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1 failed because of error code 100 and error message is: general exception >2013-03-05 13:54:16,943 WARN [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-9) [2e2c73e1] CanDoAction of action AddNFSStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ADD,ACTION_TYPE_FAILED_STORAGE_CONNECTION_NOT_EXIST >2013-03-05 13:54:17,072 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [2c2114f0] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-05 13:54:17,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [2c2114f0] START, DisconnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7cc3945a >2013-03-05 13:54:22,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [2c2114f0] FINISH, DisconnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=477}, log id: 7cc3945a >2013-03-05 13:54:54,287 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-05 13:54:54,300 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:54:54,301 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:55:09,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-2) [49f7eea3] START, ValidateStorageServerConnectionVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 270a7bfd >2013-03-05 13:55:09,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-2) [49f7eea3] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 270a7bfd >2013-03-05 13:55:09,282 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-2) [49f7eea3] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-05 13:55:09,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [49f7eea3] START, ConnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3ca181d2 >2013-03-05 13:55:09,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [49f7eea3] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=100}, log id: 3ca181d2 >2013-03-05 13:55:09,364 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (ajp-/127.0.0.1:8702-2) [49f7eea3] The connection with details lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1 failed because of error code 100 and error message is: general exception >2013-03-05 13:55:09,462 WARN [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-7) [59a21a68] CanDoAction of action AddNFSStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ADD,ACTION_TYPE_FAILED_STORAGE_CONNECTION_NOT_EXIST >2013-03-05 13:55:09,591 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-2) [6c9ee6dc] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-05 13:55:09,598 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [6c9ee6dc] START, DisconnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 280d3010 >2013-03-05 13:55:13,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [6c9ee6dc] FINISH, DisconnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=477}, log id: 280d3010 >2013-03-05 13:55:18,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-7) [35891a5d] START, ValidateStorageServerConnectionVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1dea4283 >2013-03-05 13:55:18,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-7) [35891a5d] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 1dea4283 >2013-03-05 13:55:18,921 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-7) [35891a5d] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-05 13:55:18,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [35891a5d] START, ConnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7b9b79ee >2013-03-05 13:55:18,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [35891a5d] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=100}, log id: 7b9b79ee >2013-03-05 13:55:18,990 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (ajp-/127.0.0.1:8702-7) [35891a5d] The connection with details lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1 failed because of error code 100 and error message is: general exception >2013-03-05 13:55:19,078 WARN [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-2) [33b20f8b] CanDoAction of action AddNFSStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ADD,ACTION_TYPE_FAILED_STORAGE_CONNECTION_NOT_EXIST >2013-03-05 13:55:19,172 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-9) [3dc0cdd] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-05 13:55:19,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [3dc0cdd] START, DisconnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: ebb3bf0 >2013-03-05 13:55:23,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [3dc0cdd] FINISH, DisconnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=477}, log id: ebb3bf0 >2013-03-05 13:55:54,288 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-05 13:55:54,306 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:55:54,307 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:56:52,483 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginAdminUserCommand internal: false. >2013-03-05 13:56:52,560 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-05 13:56:52,575 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:56:52,575 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:56:53,729 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (WebAdminUpperRightButtonURL) with null version, using default general for version >2013-03-05 13:56:54,285 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-05 13:56:54,297 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:56:54,298 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:57:53,557 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 13:57:53,575 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:57:53,576 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:57:54,289 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-05 13:57:54,308 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:57:54,309 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:58:52,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-4) [308af5ae] START, ValidateStorageServerConnectionVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: b4e6138 >2013-03-05 13:58:52,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-4) [308af5ae] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: b4e6138 >2013-03-05 13:58:52,964 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-4) [308af5ae] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-05 13:58:52,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-4) [308af5ae] START, ConnectStorageServerVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 14044335 >2013-03-05 13:58:53,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-4) [308af5ae] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=100}, log id: 14044335 >2013-03-05 13:58:53,156 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (ajp-/127.0.0.1:8702-4) [308af5ae] The connection with details lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1 failed because of error code 100 and error message is: general exception >2013-03-05 13:58:53,404 WARN [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [5c9d0ef3] CanDoAction of action AddNFSStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ADD,ACTION_TYPE_FAILED_STORAGE_CONNECTION_NOT_EXIST >2013-03-05 13:58:53,602 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-5) [16300d68] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-05 13:58:53,603 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 13:58:53,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [16300d68] START, DisconnectStorageServerVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 68d8f07 >2013-03-05 13:58:53,616 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:58:53,617 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:58:54,289 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-05 13:58:54,307 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:58:54,309 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:58:57,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [16300d68] FINISH, DisconnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=477}, log id: 68d8f07 >2013-03-05 13:59:53,606 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 13:59:53,638 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:59:53,639 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 13:59:54,293 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 13:59:54,316 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 13:59:54,317 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:00:53,590 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-05 14:00:53,605 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:00:53,606 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:00:54,295 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-05 14:00:54,309 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:00:54,311 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:01:39,347 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-9) [f2723a0] Running command: LogoutUserCommand internal: false. >2013-03-05 14:01:54,325 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-05 14:01:54,345 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:01:54,346 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:02:54,297 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 14:02:54,311 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:02:54,312 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:03:54,298 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-05 14:03:54,317 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:03:54,318 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:04:54,300 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 14:04:54,320 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:04:54,321 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:05:54,299 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-05 14:05:54,318 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:05:54,319 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:06:54,307 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-05 14:06:54,323 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:06:54,325 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:07:54,294 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 14:07:54,310 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:07:54,311 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:08:54,304 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-05 14:08:54,312 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:08:54,318 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:09:54,313 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 14:09:54,329 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:09:54,331 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:10:54,313 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 14:10:54,333 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:10:54,335 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:11:54,308 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-05 14:11:54,318 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:11:54,320 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:12:54,312 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-05 14:12:54,332 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:12:54,333 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:13:54,314 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-05 14:13:54,332 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:13:54,334 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:14:54,317 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-05 14:14:54,331 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:14:54,332 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:15:54,318 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 14:15:54,338 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:15:54,339 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:16:02,215 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-3) Start refreshing all users data >2013-03-05 14:16:02,383 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf. >2013-03-05 14:16:54,320 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-05 14:16:54,341 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:16:54,342 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:17:54,321 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 14:17:54,335 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:17:54,336 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:18:54,323 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-05 14:18:54,344 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:18:54,345 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:19:54,326 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-05 14:19:54,352 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:19:54,352 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:20:54,325 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-05 14:20:54,346 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:20:54,347 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:21:54,329 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 14:21:54,342 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:21:54,343 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:22:54,330 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-05 14:22:54,349 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:22:54,350 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:23:54,332 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-05 14:23:54,349 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:23:54,350 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:24:54,333 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-05 14:24:54,342 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:24:54,342 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:25:54,335 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 14:25:54,351 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:25:54,353 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:26:54,337 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 14:26:54,359 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:26:54,360 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:27:54,337 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-05 14:27:54,348 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:27:54,349 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:28:54,340 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 14:28:54,356 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:28:54,357 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:29:54,338 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 14:29:54,352 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:29:54,353 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:30:54,343 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-05 14:30:54,360 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:30:54,362 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:31:54,341 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-05 14:31:54,358 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:31:54,359 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:32:54,345 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-05 14:32:54,367 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:32:54,367 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:33:54,344 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 14:33:54,353 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:33:54,354 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:34:54,352 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-05 14:34:54,359 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:34:54,360 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:35:54,346 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-05 14:35:54,356 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:35:54,357 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:36:54,352 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 14:36:54,369 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:36:54,370 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:37:54,349 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-05 14:37:54,358 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:37:54,358 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:38:54,355 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-05 14:38:54,369 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:38:54,370 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:39:54,356 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-05 14:39:54,370 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:39:54,371 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:40:47,125 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginAdminUserCommand internal: false. >2013-03-05 14:40:47,249 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 14:40:47,261 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:40:47,262 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:40:48,916 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (WebAdminUpperRightButtonURL) with null version, using default general for version >2013-03-05 14:40:54,356 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-05 14:40:54,369 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:40:54,370 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:41:13,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-3) [73144496] START, ValidateStorageServerConnectionVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4d5c28f0 >2013-03-05 14:41:13,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-3) [73144496] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 4d5c28f0 >2013-03-05 14:41:13,572 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-3) [73144496] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-05 14:41:13,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-3) [73144496] START, ConnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6388e66f >2013-03-05 14:41:13,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-3) [73144496] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 6388e66f >2013-03-05 14:41:13,856 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-2) [6cdb7808] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-05 14:41:13,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-2) [6cdb7808] START, CreateStorageDomainVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storageDomain=org.ovirt.engine.core.common.businessentities.StorageDomainStatic@17a7f35c, args=lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1), log id: 9e43d8e >2013-03-05 14:41:18,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-2) [6cdb7808] FINISH, CreateStorageDomainVDSCommand, log id: 9e43d8e >2013-03-05 14:41:18,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-2) [6cdb7808] START, GetStorageDomainStatsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storageDomainId=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126), log id: 27e8d275 >2013-03-05 14:41:18,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-2) [6cdb7808] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage_domains@b4df509e, log id: 27e8d275 >2013-03-05 14:41:18,309 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [4e4d8828] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-05 14:41:18,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [4e4d8828] START, DisconnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5f75d2b6 >2013-03-05 14:41:22,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [4e4d8828] FINISH, DisconnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 5f75d2b6 >2013-03-05 14:41:22,440 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-1) [6cb57268] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-05 14:41:22,465 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-1) [2b0189df] Lock Acquired to object EngineLock [exclusiveLocks= key: 5849b030-626e-47cb-ad90-3ce782d831b3 value: POOL >, sharedLocks= ] >2013-03-05 14:41:22,470 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-1) [2b0189df] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-05 14:41:22,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-1) [40cc5d1c] START, ValidateStorageServerConnectionVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 4e470cfe-b114-4fb0-8f52-fc538d2b740b, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5945fe23 >2013-03-05 14:41:22,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-1) [40cc5d1c] FINISH, ValidateStorageServerConnectionVDSCommand, return: {4e470cfe-b114-4fb0-8f52-fc538d2b740b=0}, log id: 5945fe23 >2013-03-05 14:41:22,664 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8702-1) [40cc5d1c] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-05 14:41:22,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [40cc5d1c] START, ConnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 4e470cfe-b114-4fb0-8f52-fc538d2b740b, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3cb23a86 >2013-03-05 14:41:22,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [40cc5d1c] FINISH, ConnectStorageServerVDSCommand, return: {4e470cfe-b114-4fb0-8f52-fc538d2b740b=0}, log id: 3cb23a86 >2013-03-05 14:41:22,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-1) [40cc5d1c] START, CreateStoragePoolVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId=5849b030-626e-47cb-ad90-3ce782d831b3, storageType=NFS, storagePoolName=Default, masterDomainId=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, domainsIdList=[05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126], masterVersion=1), log id: 5b8af06e >2013-03-05 14:41:45,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-1) [40cc5d1c] FINISH, CreateStoragePoolVDSCommand, log id: 5b8af06e >2013-03-05 14:41:45,934 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-1) [40cc5d1c] Lock freed to object EngineLock [exclusiveLocks= key: 5849b030-626e-47cb-ad90-3ce782d831b3 value: POOL >, sharedLocks= ] >2013-03-05 14:41:45,943 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [3cc25766] Lock Acquired to object EngineLock [exclusiveLocks= key: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 value: STORAGE >, sharedLocks= ] >2013-03-05 14:41:45,952 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [3cc25766] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-05 14:41:45,981 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [3cc25766] Lock freed to object EngineLock [exclusiveLocks= key: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 value: STORAGE >, sharedLocks= ] >2013-03-05 14:41:45,983 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [3cc25766] ActivateStorage Domain. Before Connect all hosts to pool. Time:3/5/13 2:41 PM >2013-03-05 14:41:46,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-3-thread-49) [6f9eaa0a] START, ValidateStorageServerConnectionVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 4e470cfe-b114-4fb0-8f52-fc538d2b740b, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2b63116f >2013-03-05 14:41:46,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-3-thread-50) [270e91a9] START, ValidateStorageServerConnectionVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 4e470cfe-b114-4fb0-8f52-fc538d2b740b, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 50735252 >2013-03-05 14:41:46,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-3-thread-50) [270e91a9] FINISH, ValidateStorageServerConnectionVDSCommand, return: {4e470cfe-b114-4fb0-8f52-fc538d2b740b=0}, log id: 50735252 >2013-03-05 14:41:46,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-3-thread-49) [6f9eaa0a] FINISH, ValidateStorageServerConnectionVDSCommand, return: {4e470cfe-b114-4fb0-8f52-fc538d2b740b=0}, log id: 2b63116f >2013-03-05 14:41:46,040 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-3-thread-50) [270e91a9] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-05 14:41:46,041 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-3-thread-49) [6f9eaa0a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-05 14:41:46,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-3-thread-50) [270e91a9] START, ConnectStorageServerVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 4e470cfe-b114-4fb0-8f52-fc538d2b740b, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 71ff30ef >2013-03-05 14:41:46,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-3-thread-49) [6f9eaa0a] START, ConnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 4e470cfe-b114-4fb0-8f52-fc538d2b740b, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3a12b8fc >2013-03-05 14:41:46,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-3-thread-49) [6f9eaa0a] FINISH, ConnectStorageServerVDSCommand, return: {4e470cfe-b114-4fb0-8f52-fc538d2b740b=0}, log id: 3a12b8fc >2013-03-05 14:41:46,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-3-thread-50) [270e91a9] FINISH, ConnectStorageServerVDSCommand, return: {4e470cfe-b114-4fb0-8f52-fc538d2b740b=0}, log id: 71ff30ef >2013-03-05 14:41:46,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [3cc25766] START, ActivateStorageDomainVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126), log id: 7b0b3c38 >2013-03-05 14:41:46,471 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (ajp-/127.0.0.1:8702-1) [3062b62e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-05 14:41:46,552 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-1) [3062b62e] hostFromVds::selectedVds - cheetah02.scl.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool Default >2013-03-05 14:41:46,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-1) [3062b62e] START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 35edce91 >2013-03-05 14:41:48,985 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-05 14:41:48,997 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:41:48,998 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:41:50,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-1) [3062b62e] FINISH, ConnectStoragePoolVDSCommand, log id: 35edce91 >2013-03-05 14:41:50,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-1) [3062b62e] hostFromVds::Connected host to pool - selectedVds - cheetah02.scl.lab.tlv.redhat.com, spmStatus Free, storage pool Default >2013-03-05 14:41:50,808 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-1) [3062b62e] starting spm on vds cheetah02.scl.lab.tlv.redhat.com, storage pool Default, prevId -1, LVER -1 >2013-03-05 14:41:50,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-1) [3062b62e] START, SpmStartVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 450c3b3f >2013-03-05 14:41:50,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-1) [3062b62e] spmStart polling started: taskId = dfba7fce-fc3e-466c-a1f1-aef8d1e78f19 >2013-03-05 14:41:54,358 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 14:41:54,376 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:41:54,377 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:42:12,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-1) [3062b62e] spmStart polling ended: taskId = dfba7fce-fc3e-466c-a1f1-aef8d1e78f19 task status = finished >2013-03-05 14:42:12,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-1) [3062b62e] spmStart polling ended, spm status: SPM >2013-03-05 14:42:12,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8702-1) [3062b62e] START, HSMClearTaskVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, taskId=dfba7fce-fc3e-466c-a1f1-aef8d1e78f19), log id: 635c3191 >2013-03-05 14:42:12,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8702-1) [3062b62e] FINISH, HSMClearTaskVDSCommand, log id: 635c3191 >2013-03-05 14:42:12,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-1) [3062b62e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@442e2911, log id: 450c3b3f >2013-03-05 14:42:12,746 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-1) [3062b62e] Initialize Irs proxy from vds: cheetah02.scl.lab.tlv.redhat.com >2013-03-05 14:42:12,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [3062b62e] FINISH, ActivateStorageDomainVDSCommand, log id: 7b0b3c38 >2013-03-05 14:42:12,807 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [3062b62e] ActivateStorage Domain. After Connect all hosts to pool. Time:3/5/13 2:42 PM >2013-03-05 14:42:12,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) [3062b62e] START, ConnectStoragePoolVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1728a7fd >2013-03-05 14:42:12,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) [3062b62e] START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: d22b3b0 >2013-03-05 14:42:12,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) [3062b62e] FINISH, ConnectStoragePoolVDSCommand, log id: d22b3b0 >2013-03-05 14:42:16,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) [3062b62e] FINISH, ConnectStoragePoolVDSCommand, log id: 1728a7fd >2013-03-05 14:42:16,932 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [3062b62e] ActivateStorage Domain. After change storage pool status in vds. Time:3/5/13 2:42 PM >2013-03-05 14:42:48,987 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 14:42:49,002 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:42:49,004 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:42:49,906 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-8) [472a7142] Lock Acquired to object EngineLock [exclusiveLocks= key: vm value: VM_NAME >, sharedLocks= ] >2013-03-05 14:42:49,971 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-8) [472a7142] Running command: AddVmFromScratchCommand internal: false. Entities affected : ID: 99408929-82cf-4dc7-a532-9d998063fa95 Type: VdsGroups >2013-03-05 14:42:50,132 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-8) [472a7142] Lock freed to object EngineLock [exclusiveLocks= key: vm value: VM_NAME >, sharedLocks= ] >2013-03-05 14:42:54,361 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 14:42:54,381 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:42:54,382 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:42:59,760 INFO [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp-/127.0.0.1:8702-6) [3ca35673] Running command: AddVmInterfaceCommand internal: false. Entities affected : ID: 96c6bf49-8c25-44a3-af10-099bb8e1404c Type: VM, ID: 00000000-0000-0000-0000-000000000009 Type: Network >2013-03-05 14:42:59,895 INFO [org.ovirt.engine.core.bll.network.vm.ActivateDeactivateVmNicCommand] (ajp-/127.0.0.1:8702-6) [48f12b93] Running command: ActivateDeactivateVmNicCommand internal: true. Entities affected : ID: 96c6bf49-8c25-44a3-af10-099bb8e1404c Type: VM >2013-03-05 14:43:35,658 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-2) [dc24e59] Lock Acquired to object EngineLock [exclusiveLocks= key: 96c6bf49-8c25-44a3-af10-099bb8e1404c value: VM_DISK_BOOT >, sharedLocks= key: 96c6bf49-8c25-44a3-af10-099bb8e1404c value: VM >] >2013-03-05 14:43:35,689 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (ajp-/127.0.0.1:8702-2) [dc24e59] Running command: AddDiskCommand internal: false. Entities affected : ID: 96c6bf49-8c25-44a3-af10-099bb8e1404c Type: VM, ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-05 14:43:35,749 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-2) [3737a493] Running command: AddImageFromScratchCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-05 14:43:35,796 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (ajp-/127.0.0.1:8702-2) [3737a493] Lock freed to object EngineLock [exclusiveLocks= key: 96c6bf49-8c25-44a3-af10-099bb8e1404c value: VM_DISK_BOOT >, sharedLocks= key: 96c6bf49-8c25-44a3-af10-099bb8e1404c value: VM >] >2013-03-05 14:43:35,802 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-2) [3737a493] START, CreateImageVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = 3.2, storageDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, imageGroupId = df5b9b21-97dc-40de-a2f8-af9a45b15430, imageSizeInBytes = 53687091200, volumeFormat = RAW, newImageId = 80fcdc5f-203c-42a2-a01f-a59807cbed26, newImageDescription = ), log id: 190d606b >2013-03-05 14:43:35,802 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-2) [3737a493] -- CreateImageVDSCommand::ExecuteIrsBrokerCommand: calling 'createVolume' with two new parameters: description and UUID >2013-03-05 14:43:35,802 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-2) [3737a493] -- createVolume parameters: > sdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 > spUUID=5849b030-626e-47cb-ad90-3ce782d831b3 > imgGUID=df5b9b21-97dc-40de-a2f8-af9a45b15430 > size=53,687,091,200 bytes > volFormat=RAW > volType=Preallocated > volUUID=80fcdc5f-203c-42a2-a01f-a59807cbed26 > descr= > srcImgGUID=00000000-0000-0000-0000-000000000000 > srcVolUUID=00000000-0000-0000-0000-000000000000 > >2013-03-05 14:43:35,916 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (ajp-/127.0.0.1:8702-2) [3737a493] FINISH, CreateImageVDSCommand, return: 80fcdc5f-203c-42a2-a01f-a59807cbed26, log id: 190d606b >2013-03-05 14:43:35,942 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (ajp-/127.0.0.1:8702-2) [3737a493] EntityAsyncTask::Adding EntityMultiAsyncTasks object for entity df5b9b21-97dc-40de-a2f8-af9a45b15430 >2013-03-05 14:43:35,945 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-2) [3737a493] EntityMultiAsyncTasks::AttachTask: Attaching task b0576197-480d-4352-8078-d65eeec71ce3 to entity df5b9b21-97dc-40de-a2f8-af9a45b15430. >2013-03-05 14:43:36,050 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ajp-/127.0.0.1:8702-2) [3737a493] Adding task b0576197-480d-4352-8078-d65eeec71ce3 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2013-03-05 14:43:36,106 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (ajp-/127.0.0.1:8702-2) [3737a493] BaseAsyncTask::StartPollingTask: Starting to poll task b0576197-480d-4352-8078-d65eeec71ce3. >2013-03-05 14:43:36,107 INFO [org.ovirt.engine.core.bll.EntityMultiAsyncTasks] (ajp-/127.0.0.1:8702-2) [3737a493] EntityMultiAsyncTasks::StartPollingTask: Current Action Type for entity df5b9b21-97dc-40de-a2f8-af9a45b15430 is AddDisk (determined by task b0576197-480d-4352-8078-d65eeec71ce3) >2013-03-05 14:43:43,241 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2013-03-05 14:43:43,296 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-64) SPMAsyncTask::PollTask: Polling task b0576197-480d-4352-8078-d65eeec71ce3 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2013-03-05 14:43:43,298 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-64) Finished polling Tasks, will poll again in 10 seconds. >2013-03-05 14:43:48,989 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 14:43:49,003 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:43:49,004 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:43:54,362 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 14:43:54,375 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:43:54,376 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:44:48,993 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 14:44:49,012 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:44:49,013 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:44:54,360 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-05 14:44:54,370 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:44:54,371 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:45:48,997 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-05 14:45:49,046 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:45:49,047 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:45:54,367 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-05 14:45:54,380 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:45:54,381 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:46:48,998 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-05 14:46:49,011 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:46:49,012 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:46:54,367 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-05 14:46:54,387 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:46:54,387 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:47:48,994 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-05 14:47:49,017 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:47:49,017 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:47:54,369 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-05 14:47:54,384 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:47:54,385 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:48:48,991 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-05 14:48:49,009 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:48:49,010 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:48:54,370 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-05 14:48:54,380 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:48:54,380 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:49:54,374 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-05 14:49:54,393 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:49:54,393 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 14:50:54,378 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-05 14:50:54,396 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-05 14:50:54,397 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-05 15:04:19,732 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-46) SPMAsyncTask::PollTask: Polling task b0576197-480d-4352-8078-d65eeec71ce3 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2013-03-05 15:04:19,741 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (QuartzScheduler_Worker-46) BaseAsyncTask::OnTaskEndSuccess: Task b0576197-480d-4352-8078-d65eeec71ce3 (Parent Command AddDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2013-03-05 15:04:19,743 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-46) EntityAsyncTask::EndActionIfNecessary: All tasks of entity df5b9b21-97dc-40de-a2f8-af9a45b15430 has ended -> executing EndAction >2013-03-05 15:04:19,744 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (QuartzScheduler_Worker-46) EntityAsyncTask::EndAction: Ending action for 1 tasks (entity ID: df5b9b21-97dc-40de-a2f8-af9a45b15430): calling EndAction for action type AddDisk. >2013-03-05 15:04:19,747 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-3-thread-49) EntityAsyncTask::EndCommandAction [within thread] context: Attempting to EndAction AddDisk, executionIndex: 0 >2013-03-05 15:04:19,813 INFO [org.ovirt.engine.core.bll.AddDiskCommand] (pool-3-thread-49) [dc24e59] Ending command successfully: org.ovirt.engine.core.bll.AddDiskCommand >2013-03-05 15:04:19,829 INFO [org.ovirt.engine.core.bll.AddImageFromScratchCommand] (pool-3-thread-49) [3737a493] Ending command successfully: org.ovirt.engine.core.bll.AddImageFromScratchCommand >2013-03-05 15:04:19,848 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-3-thread-49) [3737a493] START, GetImageInfoVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, imageGroupId = df5b9b21-97dc-40de-a2f8-af9a45b15430, imageId = 80fcdc5f-203c-42a2-a01f-a59807cbed26), log id: 70ef8f65 >2013-03-05 15:04:19,893 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-3-thread-49) [3737a493] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@7ce6c394, log id: 70ef8f65 >2013-03-05 15:04:19,968 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-3-thread-49) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk completed, handling the result. >2013-03-05 15:04:19,969 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-3-thread-49) EntityAsyncTask::HandleEndActionResult [within thread]: EndAction for action type AddDisk succeeded, clearing tasks. >2013-03-05 15:04:19,972 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-3-thread-49) SPMAsyncTask::ClearAsyncTask: Attempting to clear task b0576197-480d-4352-8078-d65eeec71ce3 >2013-03-05 15:04:19,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-3-thread-49) START, SPMClearTaskVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, taskId = b0576197-480d-4352-8078-d65eeec71ce3), log id: 5edb4819 >2013-03-05 15:04:19,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-3-thread-49) START, HSMClearTaskVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, taskId=b0576197-480d-4352-8078-d65eeec71ce3), log id: 5810a9b6 >2013-03-05 15:04:20,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-3-thread-49) FINISH, HSMClearTaskVDSCommand, log id: 5810a9b6 >2013-03-05 15:04:20,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-3-thread-49) FINISH, SPMClearTaskVDSCommand, log id: 5edb4819 >2013-03-05 15:04:20,021 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-3-thread-49) BaseAsyncTask::RemoveTaskFromDB: Removed task b0576197-480d-4352-8078-d65eeec71ce3 from DataBase >2013-03-05 15:04:20,021 INFO [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-3-thread-49) EntityAsyncTask::HandleEndActionResult [within thread]: Removing EntityMultiAsyncTasks object for entity df5b9b21-97dc-40de-a2f8-af9a45b15430 >2013-03-05 15:05:32,689 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) Setting new tasks map. The map contains now 0 tasks >2013-03-05 15:05:32,690 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) Cleared all tasks of pool 5849b030-626e-47cb-ad90-3ce782d831b3. >2013-03-05 15:05:32,698 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-27) Could not find vds that is spm and non-operational. >2013-03-05 15:16:02,222 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-31) Start refreshing all users data >2013-03-05 15:16:02,387 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2013-03-05 15:16:02,391 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) Attempting to update VM OVFs in Data Center Default >2013-03-05 15:16:02,577 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (QuartzScheduler_Worker-28) START, UpdateVMVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 00000000-0000-0000-0000-000000000000, infoDictionary.size = 1), log id: 4087f362 >2013-03-05 15:16:02,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (QuartzScheduler_Worker-28) FINISH, UpdateVMVDSCommand, log id: 4087f362 >2013-03-05 15:16:02,653 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) Successfully updated VM OVFs in Data Center Default >2013-03-05 15:16:02,653 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) Attempting to update template OVFs in Data Center Default >2013-03-05 15:16:02,666 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) Succesfully updated templates OVFs in Data Center Default >2013-03-05 15:16:02,666 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-05 15:16:02,676 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-05 16:16:02,229 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-54) Start refreshing all users data >2013-03-05 16:16:02,677 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2013-03-05 16:16:02,680 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-55) Attempting to update VM OVFs in Data Center Default >2013-03-05 16:16:02,695 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-55) Successfully updated VM OVFs in Data Center Default >2013-03-05 16:16:02,695 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-55) Attempting to update template OVFs in Data Center Default >2013-03-05 16:16:02,698 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-55) Succesfully updated templates OVFs in Data Center Default >2013-03-05 16:16:02,699 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-55) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-05 16:16:02,701 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-55) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-05 16:36:36,618 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-42) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 : cheetah01.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-05 16:36:39,643 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-40) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 : cheetah01.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-05 16:36:42,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: 53b1939b >2013-03-05 16:36:42,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) FINISH, GetHardwareInfoVDSCommand, log id: 53b1939b >2013-03-05 16:36:42,913 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-46) [1620e680] Running command: InitVdsOnUpCommand internal: true. >2013-03-05 16:36:42,933 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-46) [5ab255b6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-05 16:36:42,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-46) [5ab255b6] START, ConnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, storageType = NFS, connectionList = [{ id: 4e470cfe-b114-4fb0-8f52-fc538d2b740b, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: a7c81cd >2013-03-05 16:36:42,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-46) [5ab255b6] FINISH, ConnectStorageServerVDSCommand, return: {4e470cfe-b114-4fb0-8f52-fc538d2b740b=0}, log id: a7c81cd >2013-03-05 16:36:42,986 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-46) [5ab255b6] Host cheetah01.scl.lab.tlv.redhat.com storage connection was succeeded >2013-03-05 16:36:43,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 442a6ac3 >2013-03-05 16:36:50,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 442a6ac3 >2013-03-05 16:36:50,434 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-47) [5ab255b6] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2013-03-05 16:36:50,452 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-46) [1b62776e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-05 16:36:50,463 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-46) [34811079] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-05 16:37:11,825 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-76) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 : cheetah01.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-05 16:37:14,850 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-75) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 : cheetah01.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-05 16:37:17,873 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-81) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 : cheetah01.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-05 16:37:20,896 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-84) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 : cheetah01.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-05 16:37:23,920 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-88) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 : cheetah01.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-05 16:37:26,942 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-95) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 : cheetah01.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-05 16:37:29,964 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-92) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 : cheetah01.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-05 16:37:32,985 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-1) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 : cheetah01.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-05 16:37:36,010 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-100) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 : cheetah01.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-05 16:37:39,032 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-3) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 : cheetah01.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-05 16:37:42,054 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-5) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 : cheetah01.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-05 16:37:45,078 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-9) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 : cheetah01.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-05 16:37:48,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: 4762156b >2013-03-05 16:37:48,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) FINISH, GetHardwareInfoVDSCommand, log id: 4762156b >2013-03-05 16:37:48,333 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-12) [59e53fd] Running command: InitVdsOnUpCommand internal: true. >2013-03-05 16:37:48,344 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-12) [1d315cca] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-05 16:37:48,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-12) [1d315cca] START, ConnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, storageType = NFS, connectionList = [{ id: 4e470cfe-b114-4fb0-8f52-fc538d2b740b, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: bf997c6 >2013-03-05 16:37:48,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-12) [1d315cca] FINISH, ConnectStorageServerVDSCommand, return: {4e470cfe-b114-4fb0-8f52-fc538d2b740b=0}, log id: bf997c6 >2013-03-05 16:37:48,411 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-12) [1d315cca] Host cheetah01.scl.lab.tlv.redhat.com storage connection was succeeded >2013-03-05 16:37:48,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 56d27ddc >2013-03-05 16:37:50,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 56d27ddc >2013-03-05 16:37:50,764 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-3-thread-50) Could not connect host cheetah01.scl.lab.tlv.redhat.com to pool Default >2013-03-05 16:37:50,812 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-12) [5ddb4012] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-05 16:37:50,814 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [5ddb4012] START, SetVdsStatusVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1d559ce4 >2013-03-05 16:37:50,816 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [5ddb4012] FINISH, SetVdsStatusVDSCommand, log id: 1d559ce4 >2013-03-05 16:37:50,825 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-12) [5ddb4012] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-05 16:37:50,866 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-12) [50128283] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-05 16:37:50,875 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-12) [257d67df] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-05 16:37:50,879 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-12) [257d67df] Host cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 : cheetah01.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-05 16:40:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) Autorecovering 1 hosts >2013-03-05 16:40:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) Autorecovering hosts id: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, name : cheetah01.scl.lab.tlv.redhat.com >2013-03-05 16:40:00,017 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [68cc5682] Lock Acquired to object EngineLock [exclusiveLocks= key: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 value: VDS >, sharedLocks= ] >2013-03-05 16:40:00,017 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [68cc5682] Running command: ActivateVdsCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-05 16:40:00,037 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [68cc5682] START, SetVdsStatusVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, status=Unassigned, nonOperationalReason=NONE), log id: 10b9534c >2013-03-05 16:40:00,041 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [68cc5682] FINISH, SetVdsStatusVDSCommand, log id: 10b9534c >2013-03-05 16:40:00,051 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [68cc5682] START, ActivateVdsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0), log id: 268864d6 >2013-03-05 16:40:00,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [68cc5682] START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: 2e653ced >2013-03-05 16:40:00,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [68cc5682] FINISH, GetHardwareInfoVDSCommand, log id: 2e653ced >2013-03-05 16:40:00,277 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [7a7ce809] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-05 16:40:00,287 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [430315c5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-05 16:40:00,292 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [430315c5] FINISH, ActivateVdsVDSCommand, log id: 268864d6 >2013-03-05 16:40:00,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-62) START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: 7c71ab84 >2013-03-05 16:40:00,349 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [430315c5] Lock freed to object EngineLock [exclusiveLocks= key: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 value: VDS >, sharedLocks= ] >2013-03-05 16:40:00,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-62) FINISH, GetHardwareInfoVDSCommand, log id: 7c71ab84 >2013-03-05 16:40:00,441 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-62) [28cf5a5e] Running command: InitVdsOnUpCommand internal: true. >2013-03-05 16:40:00,451 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-62) [50781ad5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-05 16:40:00,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-62) [50781ad5] START, ConnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, storageType = NFS, connectionList = [{ id: 4e470cfe-b114-4fb0-8f52-fc538d2b740b, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 52a77457 >2013-03-05 16:40:00,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-62) [50781ad5] FINISH, ConnectStorageServerVDSCommand, return: {4e470cfe-b114-4fb0-8f52-fc538d2b740b=0}, log id: 52a77457 >2013-03-05 16:40:00,503 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-62) [50781ad5] Host cheetah01.scl.lab.tlv.redhat.com storage connection was succeeded >2013-03-05 16:40:00,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 11e887c8 >2013-03-05 16:40:03,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 11e887c8 >2013-03-05 16:40:03,779 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-50) [50781ad5] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2013-03-05 16:40:03,790 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-62) [7d70996] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-05 16:40:03,796 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-62) [57199f7e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-05 17:16:02,236 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-76) Start refreshing all users data >2013-03-05 17:16:02,701 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2013-03-05 17:16:02,704 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-74) Attempting to update VM OVFs in Data Center Default >2013-03-05 17:16:02,706 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-74) Successfully updated VM OVFs in Data Center Default >2013-03-05 17:16:02,707 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-74) Attempting to update template OVFs in Data Center Default >2013-03-05 17:16:02,709 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-74) Succesfully updated templates OVFs in Data Center Default >2013-03-05 17:16:02,709 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-74) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-05 17:16:02,711 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-74) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-05 18:16:02,243 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-94) [52ef70c8] Start refreshing all users data >2013-03-05 18:16:02,713 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2013-03-05 18:16:02,717 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) Attempting to update VM OVFs in Data Center Default >2013-03-05 18:16:02,735 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) Successfully updated VM OVFs in Data Center Default >2013-03-05 18:16:02,735 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) Attempting to update template OVFs in Data Center Default >2013-03-05 18:16:02,739 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) Succesfully updated templates OVFs in Data Center Default >2013-03-05 18:16:02,739 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-05 18:16:02,743 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-05 19:16:02,250 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-25) Start refreshing all users data >2013-03-05 19:16:02,744 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2013-03-05 19:16:02,747 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) Attempting to update VM OVFs in Data Center Default >2013-03-05 19:16:02,750 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) Successfully updated VM OVFs in Data Center Default >2013-03-05 19:16:02,750 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) Attempting to update template OVFs in Data Center Default >2013-03-05 19:16:02,753 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) Succesfully updated templates OVFs in Data Center Default >2013-03-05 19:16:02,753 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-05 19:16:02,755 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-05 20:16:02,258 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-47) Start refreshing all users data >2013-03-05 20:16:02,756 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2013-03-05 20:16:02,760 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) Attempting to update VM OVFs in Data Center Default >2013-03-05 20:16:02,763 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) Successfully updated VM OVFs in Data Center Default >2013-03-05 20:16:02,763 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) Attempting to update template OVFs in Data Center Default >2013-03-05 20:16:02,765 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) Succesfully updated templates OVFs in Data Center Default >2013-03-05 20:16:02,765 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-05 20:16:02,768 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-05 21:16:02,265 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-72) Start refreshing all users data >2013-03-05 21:16:02,769 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2013-03-05 21:16:02,773 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-74) Attempting to update VM OVFs in Data Center Default >2013-03-05 21:16:02,792 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-74) Successfully updated VM OVFs in Data Center Default >2013-03-05 21:16:02,792 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-74) Attempting to update template OVFs in Data Center Default >2013-03-05 21:16:02,800 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-74) Succesfully updated templates OVFs in Data Center Default >2013-03-05 21:16:02,800 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-74) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-05 21:16:02,803 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-74) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-05 22:16:02,270 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-3) Start refreshing all users data >2013-03-05 22:16:02,803 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2013-03-05 22:16:02,807 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-10) Attempting to update VM OVFs in Data Center Default >2013-03-05 22:16:02,810 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-10) Successfully updated VM OVFs in Data Center Default >2013-03-05 22:16:02,810 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-10) Attempting to update template OVFs in Data Center Default >2013-03-05 22:16:02,813 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-10) Succesfully updated templates OVFs in Data Center Default >2013-03-05 22:16:02,813 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-10) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-05 22:16:02,816 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-10) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-05 23:16:02,282 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-23) Start refreshing all users data >2013-03-05 23:16:02,816 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2013-03-05 23:16:02,820 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-25) Attempting to update VM OVFs in Data Center Default >2013-03-05 23:16:02,839 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-25) Successfully updated VM OVFs in Data Center Default >2013-03-05 23:16:02,839 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-25) Attempting to update template OVFs in Data Center Default >2013-03-05 23:16:02,843 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-25) Succesfully updated templates OVFs in Data Center Default >2013-03-05 23:16:02,843 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-25) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-05 23:16:02,844 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-25) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-06 00:16:02,288 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-52) Start refreshing all users data >2013-03-06 00:16:02,845 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) Attempting to update VMs/Templates Ovf. >2013-03-06 00:16:02,848 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) Attempting to update VM OVFs in Data Center Default >2013-03-06 00:16:02,852 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) Successfully updated VM OVFs in Data Center Default >2013-03-06 00:16:02,852 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) Attempting to update template OVFs in Data Center Default >2013-03-06 00:16:02,854 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) Succesfully updated templates OVFs in Data Center Default >2013-03-06 00:16:02,855 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-06 00:16:02,857 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-06 01:16:02,293 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-70) Start refreshing all users data >2013-03-06 01:16:02,858 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2013-03-06 01:16:02,862 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-83) Attempting to update VM OVFs in Data Center Default >2013-03-06 01:16:02,865 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-83) Successfully updated VM OVFs in Data Center Default >2013-03-06 01:16:02,865 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-83) Attempting to update template OVFs in Data Center Default >2013-03-06 01:16:02,868 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-83) Succesfully updated templates OVFs in Data Center Default >2013-03-06 01:16:02,868 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-83) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-06 01:16:02,870 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-83) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-06 02:16:02,300 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-10) Start refreshing all users data >2013-03-06 02:16:02,871 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2013-03-06 02:16:02,874 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Attempting to update VM OVFs in Data Center Default >2013-03-06 02:16:02,877 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Successfully updated VM OVFs in Data Center Default >2013-03-06 02:16:02,878 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Attempting to update template OVFs in Data Center Default >2013-03-06 02:16:02,880 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Succesfully updated templates OVFs in Data Center Default >2013-03-06 02:16:02,880 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-06 02:16:02,882 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-06 03:16:02,303 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-28) Start refreshing all users data >2013-03-06 03:16:02,883 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf. >2013-03-06 03:16:02,887 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-31) Attempting to update VM OVFs in Data Center Default >2013-03-06 03:16:02,890 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-31) Successfully updated VM OVFs in Data Center Default >2013-03-06 03:16:02,890 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-31) Attempting to update template OVFs in Data Center Default >2013-03-06 03:16:02,892 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-31) Succesfully updated templates OVFs in Data Center Default >2013-03-06 03:16:02,892 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-31) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-06 03:16:02,895 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-31) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-06 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (QuartzScheduler_Worker-33) AuditLogCleanupManager::deleteAgedOutAuditLogs - entered >2013-03-06 04:16:02,308 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-56) Start refreshing all users data >2013-03-06 04:16:02,895 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2013-03-06 04:16:02,898 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) Attempting to update VM OVFs in Data Center Default >2013-03-06 04:16:02,901 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) Successfully updated VM OVFs in Data Center Default >2013-03-06 04:16:02,901 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) Attempting to update template OVFs in Data Center Default >2013-03-06 04:16:02,904 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) Succesfully updated templates OVFs in Data Center Default >2013-03-06 04:16:02,904 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-06 04:16:02,906 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-06 05:16:02,315 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-83) Start refreshing all users data >2013-03-06 05:16:02,907 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2013-03-06 05:16:02,910 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) Attempting to update VM OVFs in Data Center Default >2013-03-06 05:16:02,913 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) Successfully updated VM OVFs in Data Center Default >2013-03-06 05:16:02,913 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) Attempting to update template OVFs in Data Center Default >2013-03-06 05:16:02,915 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) Succesfully updated templates OVFs in Data Center Default >2013-03-06 05:16:02,916 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-06 05:16:02,918 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-06 06:16:02,322 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-94) [52ef70c8] Start refreshing all users data >2013-03-06 06:16:02,918 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2013-03-06 06:16:02,922 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Attempting to update VM OVFs in Data Center Default >2013-03-06 06:16:02,925 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Successfully updated VM OVFs in Data Center Default >2013-03-06 06:16:02,925 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Attempting to update template OVFs in Data Center Default >2013-03-06 06:16:02,928 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Succesfully updated templates OVFs in Data Center Default >2013-03-06 06:16:02,928 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-06 06:16:02,930 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-06 07:16:02,328 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-16) Start refreshing all users data >2013-03-06 07:16:02,930 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2013-03-06 07:16:02,933 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) Attempting to update VM OVFs in Data Center Default >2013-03-06 07:16:02,937 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) Successfully updated VM OVFs in Data Center Default >2013-03-06 07:16:02,937 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) Attempting to update template OVFs in Data Center Default >2013-03-06 07:16:02,939 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) Succesfully updated templates OVFs in Data Center Default >2013-03-06 07:16:02,939 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-06 07:16:02,943 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-06 08:16:02,335 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-57) Start refreshing all users data >2013-03-06 08:16:02,945 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf. >2013-03-06 08:16:02,949 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-48) Attempting to update VM OVFs in Data Center Default >2013-03-06 08:16:02,967 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-48) Successfully updated VM OVFs in Data Center Default >2013-03-06 08:16:02,967 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-48) Attempting to update template OVFs in Data Center Default >2013-03-06 08:16:02,980 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-48) Succesfully updated templates OVFs in Data Center Default >2013-03-06 08:16:02,980 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-48) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-06 08:16:02,982 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-48) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-06 09:16:02,343 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-83) Start refreshing all users data >2013-03-06 09:16:02,982 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2013-03-06 09:16:02,985 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) Attempting to update VM OVFs in Data Center Default >2013-03-06 09:16:02,988 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) Successfully updated VM OVFs in Data Center Default >2013-03-06 09:16:02,989 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) Attempting to update template OVFs in Data Center Default >2013-03-06 09:16:02,991 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) Succesfully updated templates OVFs in Data Center Default >2013-03-06 09:16:02,991 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-06 09:16:02,993 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-06 10:16:02,349 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-4) Start refreshing all users data >2013-03-06 10:16:02,993 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2013-03-06 10:16:02,996 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Attempting to update VM OVFs in Data Center Default >2013-03-06 10:16:02,999 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Successfully updated VM OVFs in Data Center Default >2013-03-06 10:16:02,999 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Attempting to update template OVFs in Data Center Default >2013-03-06 10:16:03,002 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Succesfully updated templates OVFs in Data Center Default >2013-03-06 10:16:03,002 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-06 10:16:03,004 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-91) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-06 11:16:02,356 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-28) Start refreshing all users data >2013-03-06 11:16:03,006 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf. >2013-03-06 11:16:03,010 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-31) Attempting to update VM OVFs in Data Center Default >2013-03-06 11:16:03,025 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-31) Successfully updated VM OVFs in Data Center Default >2013-03-06 11:16:03,025 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-31) Attempting to update template OVFs in Data Center Default >2013-03-06 11:16:03,029 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-31) Succesfully updated templates OVFs in Data Center Default >2013-03-06 11:16:03,029 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-31) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-06 11:16:03,031 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-31) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-06 12:16:02,363 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-42) Start refreshing all users data >2013-03-06 12:16:03,032 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2013-03-06 12:16:03,036 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) Attempting to update VM OVFs in Data Center Default >2013-03-06 12:16:03,039 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) Successfully updated VM OVFs in Data Center Default >2013-03-06 12:16:03,039 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) Attempting to update template OVFs in Data Center Default >2013-03-06 12:16:03,041 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) Succesfully updated templates OVFs in Data Center Default >2013-03-06 12:16:03,042 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-06 12:16:03,044 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-06 13:16:02,369 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-68) Start refreshing all users data >2013-03-06 13:16:03,045 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf. >2013-03-06 13:16:03,048 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-78) Attempting to update VM OVFs in Data Center Default >2013-03-06 13:16:03,052 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-78) Successfully updated VM OVFs in Data Center Default >2013-03-06 13:16:03,052 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-78) Attempting to update template OVFs in Data Center Default >2013-03-06 13:16:03,054 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-78) Succesfully updated templates OVFs in Data Center Default >2013-03-06 13:16:03,054 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-78) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-06 13:16:03,057 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-78) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-06 14:16:02,375 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-94) [52ef70c8] Start refreshing all users data >2013-03-06 14:16:03,057 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2013-03-06 14:16:03,060 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-95) Attempting to update VM OVFs in Data Center Default >2013-03-06 14:16:03,063 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-95) Successfully updated VM OVFs in Data Center Default >2013-03-06 14:16:03,064 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-95) Attempting to update template OVFs in Data Center Default >2013-03-06 14:16:03,066 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-95) Succesfully updated templates OVFs in Data Center Default >2013-03-06 14:16:03,066 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-95) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-06 14:16:03,068 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-95) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-06 15:16:02,381 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-16) Start refreshing all users data >2013-03-06 15:16:03,068 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [4bbfe1d7] Attempting to update VMs/Templates Ovf. >2013-03-06 15:16:03,070 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [4bbfe1d7] Attempting to update VM OVFs in Data Center Default >2013-03-06 15:16:03,072 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [4bbfe1d7] Successfully updated VM OVFs in Data Center Default >2013-03-06 15:16:03,073 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [4bbfe1d7] Attempting to update template OVFs in Data Center Default >2013-03-06 15:16:03,074 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [4bbfe1d7] Succesfully updated templates OVFs in Data Center Default >2013-03-06 15:16:03,074 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [4bbfe1d7] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-06 15:16:03,075 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [4bbfe1d7] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-06 16:16:02,388 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-44) Start refreshing all users data >2013-03-06 16:16:03,075 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2013-03-06 16:16:03,079 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) Attempting to update VM OVFs in Data Center Default >2013-03-06 16:16:03,081 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) Successfully updated VM OVFs in Data Center Default >2013-03-06 16:16:03,082 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) Attempting to update template OVFs in Data Center Default >2013-03-06 16:16:03,084 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) Succesfully updated templates OVFs in Data Center Default >2013-03-06 16:16:03,084 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-06 16:16:03,087 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-06 17:07:51,957 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-34) Timer update runtimeinfo failed. Exception:: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) [engine-dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58) [engine-dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:37) [engine-dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:32) [engine-dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:202) [engine-vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor106.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [engine-scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz-2.1.2.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz-2.1.2.jar:] >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:147) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > ... 20 more >Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:390) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139) > ... 22 more >Caused by: javax.resource.ResourceException: IJ000658: Unexpected throwable while trying to create a connection: null > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:378) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:397) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:365) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329) > ... 25 more >Caused by: javax.resource.ResourceException: Could not create connection > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:282) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:240) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.createConnectionEventListener(SemaphoreArrayListManagedConnectionPool.java:775) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:345) > ... 28 more >Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136) > at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66) > at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125) > at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30) > at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22) > at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) > at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) > at org.postgresql.Driver.makeConnection(Driver.java:393) > at org.postgresql.Driver.connect(Driver.java:267) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:254) > ... 31 more >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_09-icedtea] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.connect(Socket.java:528) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.<init>(Socket.java:425) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.<init>(Socket.java:208) [rt.jar:1.7.0_09-icedtea] > at org.postgresql.core.PGStream.<init>(PGStream.java:62) > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:76) > ... 40 more > >2013-03-06 17:07:52,035 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-17) Timer update runtimeinfo failed. Exception:: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) [engine-dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58) [engine-dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:37) [engine-dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:32) [engine-dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:202) [engine-vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor106.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [engine-scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz-2.1.2.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz-2.1.2.jar:] >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:147) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > ... 20 more >Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:390) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139) > ... 22 more >Caused by: javax.resource.ResourceException: IJ000658: Unexpected throwable while trying to create a connection: null > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:378) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:397) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:365) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329) > ... 25 more >Caused by: javax.resource.ResourceException: Could not create connection > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:282) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:240) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.createConnectionEventListener(SemaphoreArrayListManagedConnectionPool.java:775) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:345) > ... 28 more >Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136) > at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66) > at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125) > at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30) > at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22) > at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) > at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) > at org.postgresql.Driver.makeConnection(Driver.java:393) > at org.postgresql.Driver.connect(Driver.java:267) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:254) > ... 31 more >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_09-icedtea] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.connect(Socket.java:528) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.<init>(Socket.java:425) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.<init>(Socket.java:208) [rt.jar:1.7.0_09-icedtea] > at org.postgresql.core.PGStream.<init>(PGStream.java:62) > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:76) > ... 40 more > >2013-03-06 17:07:52,804 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (QuartzScheduler_Worker-37) failed to invoke sceduled method refreshLightWeightData: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [engine-scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz-2.1.2.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz-2.1.2.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) [engine-dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) [engine-dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:96) [engine-dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:89) [engine-dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterManager.refreshLightWeightData(GlusterManager.java:165) [engine-bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:147) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > ... 19 more >Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:390) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139) > ... 21 more >Caused by: javax.resource.ResourceException: IJ000658: Unexpected throwable while trying to create a connection: null > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:378) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:397) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:365) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329) > ... 24 more >Caused by: javax.resource.ResourceException: Could not create connection > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:282) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:240) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.createConnectionEventListener(SemaphoreArrayListManagedConnectionPool.java:775) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:345) > ... 27 more >Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136) > at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66) > at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125) > at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30) > at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22) > at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) > at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) > at org.postgresql.Driver.makeConnection(Driver.java:393) > at org.postgresql.Driver.connect(Driver.java:267) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:254) > ... 30 more >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_09-icedtea] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.connect(Socket.java:528) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.<init>(Socket.java:425) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.<init>(Socket.java:208) [rt.jar:1.7.0_09-icedtea] > at org.postgresql.core.PGStream.<init>(PGStream.java:62) > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:76) > ... 39 more > >2013-03-06 17:07:54,978 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-22) Timer update runtimeinfo failed. Exception:: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) [engine-dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58) [engine-dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:37) [engine-dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:32) [engine-dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:202) [engine-vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor106.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [engine-scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz-2.1.2.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz-2.1.2.jar:] >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:147) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > ... 20 more >Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:390) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139) > ... 22 more >Caused by: javax.resource.ResourceException: IJ000658: Unexpected throwable while trying to create a connection: null > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:378) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:397) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:365) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329) > ... 25 more >Caused by: javax.resource.ResourceException: Could not create connection > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:282) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:240) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.createConnectionEventListener(SemaphoreArrayListManagedConnectionPool.java:775) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:345) > ... 28 more >Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136) > at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66) > at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125) > at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30) > at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22) > at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) > at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) > at org.postgresql.Driver.makeConnection(Driver.java:393) > at org.postgresql.Driver.connect(Driver.java:267) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:254) > ... 31 more >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_09-icedtea] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.connect(Socket.java:528) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.<init>(Socket.java:425) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.<init>(Socket.java:208) [rt.jar:1.7.0_09-icedtea] > at org.postgresql.core.PGStream.<init>(PGStream.java:62) > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:76) > ... 40 more > >2013-03-06 17:07:55,050 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-21) Timer update runtimeinfo failed. Exception:: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) [engine-dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58) [engine-dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:37) [engine-dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:32) [engine-dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:202) [engine-vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor106.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [engine-scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz-2.1.2.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz-2.1.2.jar:] >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:147) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > ... 20 more >Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:390) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139) > ... 22 more >Caused by: javax.resource.ResourceException: IJ000658: Unexpected throwable while trying to create a connection: null > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:378) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:397) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:365) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329) > ... 25 more >Caused by: javax.resource.ResourceException: Could not create connection > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:282) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:240) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.createConnectionEventListener(SemaphoreArrayListManagedConnectionPool.java:775) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:345) > ... 28 more >Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136) > at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66) > at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125) > at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30) > at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22) > at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) > at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) > at org.postgresql.Driver.makeConnection(Driver.java:393) > at org.postgresql.Driver.connect(Driver.java:267) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:254) > ... 31 more >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_09-icedtea] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.connect(Socket.java:528) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.<init>(Socket.java:425) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.<init>(Socket.java:208) [rt.jar:1.7.0_09-icedtea] > at org.postgresql.core.PGStream.<init>(PGStream.java:62) > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:76) > ... 40 more > >2013-03-06 17:07:57,839 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (QuartzScheduler_Worker-23) failed to invoke sceduled method refreshLightWeightData: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [engine-scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz-2.1.2.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz-2.1.2.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) [engine-dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) [engine-dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:96) [engine-dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:89) [engine-dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterManager.refreshLightWeightData(GlusterManager.java:165) [engine-bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:147) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > ... 19 more >Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:390) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139) > ... 21 more >Caused by: javax.resource.ResourceException: IJ000658: Unexpected throwable while trying to create a connection: null > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:378) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:397) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:365) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329) > ... 24 more >Caused by: javax.resource.ResourceException: Could not create connection > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:282) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:240) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.createConnectionEventListener(SemaphoreArrayListManagedConnectionPool.java:775) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:345) > ... 27 more >Caused by: org.postgresql.util.PSQLException: The connection attempt failed. > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:150) > at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66) > at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125) > at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30) > at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22) > at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) > at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) > at org.postgresql.Driver.makeConnection(Driver.java:393) > at org.postgresql.Driver.connect(Driver.java:267) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:254) > ... 30 more >Caused by: java.net.SocketException: Network is unreachable > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_09-icedtea] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.connect(Socket.java:528) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.<init>(Socket.java:425) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.<init>(Socket.java:208) [rt.jar:1.7.0_09-icedtea] > at org.postgresql.core.PGStream.<init>(PGStream.java:62) > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:76) > ... 39 more > >2013-03-06 17:07:57,995 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-100) Timer update runtimeinfo failed. Exception:: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) [engine-dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58) [engine-dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:37) [engine-dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:32) [engine-dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:202) [engine-vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor106.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [engine-scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz-2.1.2.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz-2.1.2.jar:] >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:147) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > ... 20 more >Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:390) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139) > ... 22 more >Caused by: javax.resource.ResourceException: IJ000658: Unexpected throwable while trying to create a connection: null > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:378) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:397) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:365) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329) > ... 25 more >Caused by: javax.resource.ResourceException: Could not create connection > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:282) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:240) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.createConnectionEventListener(SemaphoreArrayListManagedConnectionPool.java:775) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:345) > ... 28 more >Caused by: org.postgresql.util.PSQLException: The connection attempt failed. > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:150) > at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66) > at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125) > at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30) > at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22) > at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) > at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) > at org.postgresql.Driver.makeConnection(Driver.java:393) > at org.postgresql.Driver.connect(Driver.java:267) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:254) > ... 31 more >Caused by: java.net.SocketException: Network is unreachable > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_09-icedtea] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.connect(Socket.java:528) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.<init>(Socket.java:425) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.<init>(Socket.java:208) [rt.jar:1.7.0_09-icedtea] > at org.postgresql.core.PGStream.<init>(PGStream.java:62) > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:76) > ... 40 more > >2013-03-06 17:07:58,068 ERROR [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-41) Timer update runtimeinfo failed. Exception:: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:155) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:121) [engine-dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:124) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:75) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:66) [engine-dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:58) [engine-dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:37) [engine-dal.jar:] > at org.ovirt.engine.core.dao.VdsDAODbFacadeImpl.get(VdsDAODbFacadeImpl.java:32) [engine-dal.jar:] > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:202) [engine-vdsbroker.jar:] > at sun.reflect.GeneratedMethodAccessor106.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [engine-scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz-2.1.2.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz-2.1.2.jar:] >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:147) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc-3.1.1.RELEASE.jar:3.1.1.RELEASE] > ... 20 more >Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:390) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139) > ... 22 more >Caused by: javax.resource.ResourceException: IJ000658: Unexpected throwable while trying to create a connection: null > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:378) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:397) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:365) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329) > ... 25 more >Caused by: javax.resource.ResourceException: Could not create connection > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:282) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:240) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.createConnectionEventListener(SemaphoreArrayListManagedConnectionPool.java:775) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:345) > ... 28 more >Caused by: org.postgresql.util.PSQLException: The connection attempt failed. > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:150) > at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66) > at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125) > at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30) > at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22) > at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) > at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) > at org.postgresql.Driver.makeConnection(Driver.java:393) > at org.postgresql.Driver.connect(Driver.java:267) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:254) > ... 31 more >Caused by: java.net.SocketException: Network is unreachable > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_09-icedtea] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_09-icedtea] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.connect(Socket.java:528) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.<init>(Socket.java:425) [rt.jar:1.7.0_09-icedtea] > at java.net.Socket.<init>(Socket.java:208) [rt.jar:1.7.0_09-icedtea] > at org.postgresql.core.PGStream.<init>(PGStream.java:62) > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:76) > ... 40 more > >2013-03-07 10:30:31,789 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 32) Start time: 3/7/13 10:30 AM >2013-03-07 10:30:33,101 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ServerService Thread Pool -- 32) Failed to decrypt Data must start with zero >2013-03-07 10:30:33,101 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ServerService Thread Pool -- 32) Failed to decrypt value for property TruststorePass will be used encrypted value >2013-03-07 10:30:33,161 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 32) VDSBrokerFrontend: 3/7/13 10:30 AM >2013-03-07 10:30:33,170 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 32) CpuFlagsManager: 3/7/13 10:30 AM >2013-03-07 10:30:33,171 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 32) Setting audit clean up manager to run at: 35 35 3 * * ? >2013-03-07 10:30:33,193 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 32) AuditLogCleanupManager: 3/7/13 10:30 AM >2013-03-07 10:30:33,197 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 32) TagsDirector initialization >2013-03-07 10:30:33,220 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 32) Tag root added to tree >2013-03-07 10:30:33,290 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 32) TagsDirector: 3/7/13 10:30 AM >2013-03-07 10:30:33,299 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 32) IsoDomainListSyncronizer: 3/7/13 10:30 AM >2013-03-07 10:30:33,337 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 32) InitHandlers: 3/7/13 10:30 AM >2013-03-07 10:30:33,355 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 32) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2013-03-07 10:30:33,356 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 32) ErrorTranslator: 3/7/13 10:30 AM >2013-03-07 10:30:33,356 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 32) VdsErrorTranslator: 3/7/13 10:30 AM >2013-03-07 10:30:33,357 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 32) Mark uncompleted jobs as UNKNOWN: 3/7/13 10:30 AM >2013-03-07 10:30:33,511 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 32) JobRepositoryCleanupManager: 3/7/13 10:30 AM >2013-03-07 10:30:33,512 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 32) AutoRecoveryManager: 3/7/13 10:30 AM >2013-03-07 10:30:33,515 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 32) ExecutionMessageDirector: 3/7/13 10:30 AM >2013-03-07 10:30:33,545 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 33) InitResourceManager: 3/7/13 10:30 AM >2013-03-07 10:30:33,550 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 33) ResourceManager::ResourceManager::Entered >2013-03-07 10:30:33,553 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (ServerService Thread Pool -- 32) UsersDomainsCacheManager: 3/7/13 10:30 AM >2013-03-07 10:30:33,620 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (ServerService Thread Pool -- 32) DbUserCacheManager: 3/7/13 10:30 AM >2013-03-07 10:30:33,670 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-1) Start refreshing all users data >2013-03-07 10:30:34,047 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 33) Eneterd VdsManager:constructor >2013-03-07 10:30:34,049 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 33) vdsBroker(cheetah01.scl.lab.tlv.redhat.com,54,321) >2013-03-07 10:30:34,382 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 33) ResourceManager::AddVds - VDS cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 was added to the Resource Manager >2013-03-07 10:30:34,387 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 33) Eneterd VdsManager:constructor >2013-03-07 10:30:34,388 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 33) vdsBroker(cheetah02.scl.lab.tlv.redhat.com,54,321) >2013-03-07 10:30:34,398 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 33) ResourceManager::AddVds - VDS 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 was added to the Resource Manager >2013-03-07 10:30:34,440 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ServerService Thread Pool -- 33) AsyncTaskManager: Initialization of AsyncTaskManager completed successfully. >2013-03-07 10:30:34,442 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 33) AsyncTaskManager: 3/7/13 10:30 AM >2013-03-07 10:30:34,449 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (ServerService Thread Pool -- 33) Initialization of OvfDataUpdater completed successfully. >2013-03-07 10:30:34,451 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 33) OvfDataUpdater: 3/7/13 10:30 AM >2013-03-07 10:30:34,453 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 33) VdsLoadBalancer: 3/7/13 10:30 AM >2013-03-07 10:30:34,465 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (pool-3-thread-1) MacPoolManager started: 3/7/13 10:30 AM >2013-03-07 10:30:34,486 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 33) Init VM Custom Properties utilities: 3/7/13 10:30 AM >2013-03-07 10:30:34,669 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (pool-3-thread-1) MacPoolManager finished: 3/7/13 10:30 AM >2013-03-07 10:30:38,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: 3eb872f5 >2013-03-07 10:30:38,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-3) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 541d644b >2013-03-07 10:30:38,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) FINISH, GetHardwareInfoVDSCommand, log id: 3eb872f5 >2013-03-07 10:30:38,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-3) FINISH, GetHardwareInfoVDSCommand, log id: 541d644b >2013-03-07 10:30:38,467 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: UNASSIGNED not exist in string table >2013-03-07 10:30:38,468 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2013-03-07 10:30:38,474 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2013-03-07 10:30:38,474 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2013-03-07 10:30:38,474 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: DWH_STOPPED not exist in string table >2013-03-07 10:30:38,474 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: DWH_STARTED not exist in string table >2013-03-07 10:30:38,475 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: DWH_ERROR not exist in string table >2013-03-07 10:30:38,475 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: EXTERNAL_EVENT_NORMAL not exist in string table >2013-03-07 10:30:38,475 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: EXTERNAL_EVENT_WARNING not exist in string table >2013-03-07 10:30:38,475 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: EXTERNAL_EVENT_ERROR not exist in string table >2013-03-07 10:30:38,475 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: EXTERNAL_ALERT not exist in string table >2013-03-07 10:30:38,484 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: USER_ATTACH_DISK_TO_VM not have severity. Assumed Normal >2013-03-07 10:30:38,485 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: USER_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2013-03-07 10:30:38,486 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: USER_FAILED_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2013-03-07 10:30:38,486 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not have severity. Assumed Normal >2013-03-07 10:30:38,488 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: VM_MIGRATION_START_SYSTEM_INITIATED not have severity. Assumed Normal >2013-03-07 10:30:38,488 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not have severity. Assumed Normal >2013-03-07 10:30:38,489 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: NETWORK_UPDATE_VM_INTERFACE_LINK_UP not have severity. Assumed Normal >2013-03-07 10:30:38,489 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) AuditLogType: NETWORK_UPDATE_VM_INTERFACE_LINK_DOWN not have severity. Assumed Normal >2013-03-07 10:30:38,760 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-2) Initializing Host: cheetah01.scl.lab.tlv.redhat.com >2013-03-07 10:30:38,755 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-3) Initializing Host: cheetah02.scl.lab.tlv.redhat.com >2013-03-07 10:30:39,067 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-3) [4f9bf40f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 10:30:39,067 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [60232fff] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-07 10:30:44,465 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-7) [1c44d063] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 10:30:44,587 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [1c44d063] hostFromVds::selectedVds - cheetah01.scl.lab.tlv.redhat.com, spmStatus Free, storage pool Default >2013-03-07 10:30:44,588 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [1c44d063] Found spm host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, host name: cheetah02.scl.lab.tlv.redhat.com, according to spmId: 2. >2013-03-07 10:30:44,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [1c44d063] SpmStatus on vds 5e9ad544-fe9f-4669-8b11-1631ba7d1c12: SPM >2013-03-07 10:30:44,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [1c44d063] Using old spm server: cheetah02.scl.lab.tlv.redhat.com, no start needed >2013-03-07 10:30:44,631 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) [1c44d063] Initialize Irs proxy from vds: cheetah02.scl.lab.tlv.redhat.com >2013-03-07 10:30:44,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 76fff184 >2013-03-07 10:30:44,949 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) -- SPMGetAllTasksInfoVDSCommand::ExecuteIrsBrokerCommand: Attempting on storage pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 10:30:44,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) START, HSMGetAllTasksInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 11fae062 >2013-03-07 10:30:44,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 11fae062 >2013-03-07 10:30:44,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (QuartzScheduler_Worker-10) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 76fff184 >2013-03-07 10:30:44,980 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (QuartzScheduler_Worker-10) Discovered no tasks on Storage Pool Default >2013-03-07 10:35:34,636 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Loaded file "/usr/share/ovirt-engine/conf/engine.conf.defaults". >2013-03-07 10:35:34,637 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Loaded file "/etc/sysconfig/ovirt-engine". >2013-03-07 10:35:34,638 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine". >2013-03-07 10:35:34,638 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_PROPERTIES" is "". >2013-03-07 10:35:34,638 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2013-03-07 10:35:34,638 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_PERM_MIN" is "256m". >2013-03-07 10:35:34,638 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "JBOSS_HOME" is "/usr/share/jbossas". >2013-03-07 10:35:34,638 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_DB_USER" is "engine". >2013-03-07 10:35:34,638 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2013-03-07 10:35:34,638 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2013-03-07 10:35:34,638 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_STOP_TIME" is "10". >2013-03-07 10:35:34,638 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_FQDN" is "gadi-rhevm.scl.lab.tlv.redhat.com". >2013-03-07 10:35:34,638 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_HEAP_MAX" is "1g". >2013-03-07 10:35:34,638 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2013-03-07 10:35:34,638 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_LOCK" is "/var/lock/ovirt-engine". >2013-03-07 10:35:34,638 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_STOP_INTERVAL" is "1". >2013-03-07 10:35:34,639 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_PROXY_ENABLED" is "true". >2013-03-07 10:35:34,639 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2013-03-07 10:35:34,639 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_APPS" is "engine.ear". >2013-03-07 10:35:34,639 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_DB_PASSWORD" is "64c5fd2979a86168". >2013-03-07 10:35:34,639 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2013-03-07 10:35:34,639 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2013-03-07 10:35:34,639 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2013-03-07 10:35:34,639 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_HTTP_PORT" is "8700". >2013-03-07 10:35:34,639 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_AJP_PORT" is "8702". >2013-03-07 10:35:34,639 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_PERM_MAX" is "256m". >2013-03-07 10:35:34,639 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_VERBOSE_GC" is "false". >2013-03-07 10:35:34,639 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_HTTPS_PORT" is "8701". >2013-03-07 10:35:34,640 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2013-03-07 10:35:34,640 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2013-03-07 10:35:34,640 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_NOFILE" is "65535". >2013-03-07 10:35:34,640 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2013-03-07 10:35:34,640 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2013-03-07 10:35:34,640 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_PID" is "/var/run/ovirt-engine.pid". >2013-03-07 10:35:34,640 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2013-03-07 10:35:34,640 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_HTTP_ENABLED" is "false". >2013-03-07 10:35:34,640 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_GROUP" is "ovirt". >2013-03-07 10:35:34,640 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "JAVA_HOME" is "/usr/lib/jvm/jre-1.7.0-openjdk.x86_64". >2013-03-07 10:35:34,640 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2013-03-07 10:35:34,640 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_DB_SSL" is "false". >2013-03-07 10:35:34,640 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2013-03-07 10:35:34,640 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_HEAP_MIN" is "1g". >2013-03-07 10:35:34,641 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_AJP_ENABLED" is "true". >2013-03-07 10:35:34,641 INFO [org.ovirt.engine.core.utils.LocalConfig] (ajp-/127.0.0.1:8702-7) Value of property "ENGINE_USER" is "ovirt". >2013-03-07 10:35:46,373 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginAdminUserCommand internal: false. >2013-03-07 10:35:46,497 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ajp-/127.0.0.1:8702-3) Failed to decrypt Data must start with zero >2013-03-07 10:35:46,498 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-3) Failed to decrypt value for property LocalAdminPassword will be used encrypted value >2013-03-07 10:35:46,541 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ajp-/127.0.0.1:8702-3) Failed to decrypt Data must start with zero >2013-03-07 10:35:46,541 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-3) Failed to decrypt value for property LocalAdminPassword will be used encrypted value >2013-03-07 10:35:46,571 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ajp-/127.0.0.1:8702-3) Failed to decrypt Data must start with zero >2013-03-07 10:35:46,571 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-3) Failed to decrypt value for property LocalAdminPassword will be used encrypted value >2013-03-07 10:35:46,596 ERROR [org.ovirt.engine.core.engineencryptutils.EncryptionUtils] (ajp-/127.0.0.1:8702-3) Failed to decrypt Data must start with zero >2013-03-07 10:35:46,597 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-3) Failed to decrypt value for property LocalAdminPassword will be used encrypted value >2013-03-07 10:35:47,272 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 10:35:47,293 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:35:47,308 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:35:48,473 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (WebAdminUpperRightButtonURL) with null version, using default general for version >2013-03-07 10:36:49,336 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 10:36:49,349 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:36:49,350 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:37:49,333 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 10:37:49,374 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:37:49,378 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:38:49,342 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 10:38:49,358 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:38:49,359 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:39:49,339 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 10:39:49,349 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:39:49,350 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:40:49,335 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 10:40:49,349 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:40:49,350 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:41:49,341 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 10:41:49,348 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:41:49,493 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:42:49,340 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 10:42:49,350 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:42:49,351 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:43:49,340 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 10:43:49,348 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:43:49,349 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:44:49,342 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 10:44:49,357 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:44:49,358 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:45:49,345 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 10:45:49,367 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:45:49,368 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:46:49,347 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 10:46:49,353 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:46:49,354 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:47:49,347 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 10:47:49,355 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:47:49,356 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:48:49,348 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 10:48:49,357 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:48:49,358 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:49:49,349 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 10:49:49,359 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:49:49,360 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:50:49,351 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 10:50:49,359 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:50:49,360 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:51:49,351 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 10:51:49,360 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:51:49,360 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:52:49,353 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 10:52:49,360 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:52:49,363 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:53:49,355 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 10:53:49,363 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:53:49,363 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:54:49,356 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 10:54:49,364 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:54:49,364 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:55:49,380 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 10:55:49,389 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:55:49,390 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:56:49,361 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 10:56:49,370 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:56:49,370 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:57:49,362 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 10:57:49,370 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:57:49,371 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:58:49,363 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 10:58:49,369 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:58:49,369 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 10:59:49,366 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 10:59:49,378 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 10:59:49,379 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 11:00:49,367 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 11:00:49,382 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 11:00:49,383 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 11:01:49,368 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 11:01:49,387 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 11:01:49,387 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 11:02:49,368 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 11:02:49,384 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 11:02:49,384 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 11:03:49,372 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 11:03:49,379 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 11:03:49,379 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 11:04:49,373 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 11:04:49,380 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 11:04:49,380 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 11:05:49,373 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 11:05:49,382 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 11:05:49,382 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 11:06:49,376 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 11:06:49,382 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 11:06:49,382 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 11:07:49,381 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 11:07:49,389 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 11:07:49,389 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 11:30:33,686 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-32) Start refreshing all users data >2013-03-07 11:30:34,450 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2013-03-07 11:30:34,455 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-37) Attempting to update VM OVFs in Data Center Default >2013-03-07 11:30:34,478 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-37) Successfully updated VM OVFs in Data Center Default >2013-03-07 11:30:34,478 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-37) Attempting to update template OVFs in Data Center Default >2013-03-07 11:30:34,511 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-37) Succesfully updated templates OVFs in Data Center Default >2013-03-07 11:30:34,511 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-37) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-07 11:30:34,514 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-37) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-07 12:30:33,688 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-76) Start refreshing all users data >2013-03-07 12:30:34,515 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2013-03-07 12:30:34,518 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) Attempting to update VM OVFs in Data Center Default >2013-03-07 12:30:34,519 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) Successfully updated VM OVFs in Data Center Default >2013-03-07 12:30:34,519 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) Attempting to update template OVFs in Data Center Default >2013-03-07 12:30:34,520 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) Succesfully updated templates OVFs in Data Center Default >2013-03-07 12:30:34,520 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-07 12:30:34,520 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-07 13:30:33,690 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-20) Start refreshing all users data >2013-03-07 13:30:34,520 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2013-03-07 13:30:34,523 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-27) Attempting to update VM OVFs in Data Center Default >2013-03-07 13:30:34,531 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-27) Successfully updated VM OVFs in Data Center Default >2013-03-07 13:30:34,531 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-27) Attempting to update template OVFs in Data Center Default >2013-03-07 13:30:34,534 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-27) Succesfully updated templates OVFs in Data Center Default >2013-03-07 13:30:34,534 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-27) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-07 13:30:34,535 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-27) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-07 13:37:03,942 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:03,950 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:11,200 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:11,206 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:11,247 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:11,282 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:11,376 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:11,383 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:11,428 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:11,436 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:11,436 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 13:37:11,445 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 13:37:12,299 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:12,305 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:12,362 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:12,369 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:12,415 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:12,425 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:12,470 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:12,478 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:12,523 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:12,530 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:12,576 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:12,582 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:12,629 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:12,635 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:12,685 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:12,692 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:12,735 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:12,742 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:12,742 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 13:37:12,853 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:12,859 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:12,905 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:12,912 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:12,954 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:12,961 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:13,005 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:13,012 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:13,066 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:13,075 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:13,142 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:13,154 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:13,220 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:13,231 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:13,280 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:13,286 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:13,337 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:13,356 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:13,399 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:13,408 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:13,456 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:13,466 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:13,514 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:13,529 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:13,578 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:13,588 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:13,724 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:13,733 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:13,775 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:13,781 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:13,826 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:13,837 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:13,886 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:13,892 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:13,936 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:13,943 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:13,985 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:13,993 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:14,050 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:14,057 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:14,104 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:14,110 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:14,155 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:14,162 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:14,205 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:14,211 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:14,255 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:14,261 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:14,305 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:14,312 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:14,356 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:14,363 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:14,405 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:14,412 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:14,456 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:14,462 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:14,506 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:14,514 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:14,589 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:14,596 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:14,645 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:14,652 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:14,699 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:14,705 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:14,746 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:14,753 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:14,796 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:14,801 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:14,849 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:14,856 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:14,905 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:14,913 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:14,956 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:14,965 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,006 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:15,011 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,056 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:15,063 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,149 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:15,155 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,205 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:15,212 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,256 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:15,263 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,306 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:15,312 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,356 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:15,362 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,406 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:15,413 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,455 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:15,463 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,505 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:15,517 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,566 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:15,574 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,624 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:15,631 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,675 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:15,682 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,723 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:15,731 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,769 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:15,775 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,816 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:15,824 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,867 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:15,874 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,916 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:15,925 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:15,925 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 13:37:16,024 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:16,033 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:16,076 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:16,084 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:16,132 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:16,144 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:16,190 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:16,197 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:16,490 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:16,497 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:16,545 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:16,552 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:16,598 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:16,605 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:16,645 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:16,654 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:16,707 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:16,715 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:16,756 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:16,766 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:37:58,112 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:37:58,121 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:01,804 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:01,813 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:01,856 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:01,884 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:01,934 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:01,941 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:01,999 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:02,006 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:02,006 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 13:38:02,387 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:02,392 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:02,434 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:02,440 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:02,484 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:02,492 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:02,542 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:02,551 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:02,595 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:02,605 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:02,658 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:02,668 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:02,715 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:02,721 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:02,764 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:02,771 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:02,815 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:02,821 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:02,821 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 13:38:02,876 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:02,885 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:02,926 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:02,933 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:02,975 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:02,981 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,025 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,036 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,083 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,090 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,136 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,143 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,190 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,195 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,237 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,243 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,285 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,293 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,335 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,345 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,389 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,395 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,436 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,442 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,486 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,493 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,538 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,546 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,595 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,602 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,645 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,652 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,696 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,702 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,745 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,751 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,796 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,802 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,845 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,851 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,894 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,900 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,945 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:03,951 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:03,996 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,002 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,046 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,052 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,097 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,105 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,146 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,153 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,196 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,203 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,244 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,251 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,295 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,304 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,378 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,385 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,425 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,430 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,475 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,483 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,524 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,532 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,576 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,582 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,626 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,632 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,676 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,681 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,725 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,731 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,776 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,781 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,825 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,832 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,874 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,881 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,925 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,931 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:04,975 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:04,982 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,026 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,031 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,075 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,082 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,126 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,132 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,175 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,185 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,226 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,232 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,277 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,283 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,325 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,333 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,376 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,384 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,426 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,434 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,476 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,485 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,526 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,536 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,576 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,584 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,626 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,634 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,634 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 13:38:05,696 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,704 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,746 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,754 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,795 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,803 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,846 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,858 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,917 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,925 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:05,966 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:05,979 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:06,027 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:06,035 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:06,075 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:06,084 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:06,126 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:06,132 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:38:06,176 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 13:38:06,183 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:40:55,753 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:40:55,761 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:40:59,316 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:40:59,321 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:40:59,458 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 13:41:41,788 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 13:41:41,794 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:41:45,286 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 13:41:45,293 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:41:58,613 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 13:41:58,619 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 13:42:02,649 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 13:42:02,656 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:17:12,564 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 14:17:12,572 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:17:16,573 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 14:17:16,579 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:18:15,009 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 14:18:15,016 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:18:18,595 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 14:18:18,603 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:23:29,316 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 14:23:29,324 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:23:33,586 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 14:23:33,592 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:23:35,289 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 14:23:38,469 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 14:23:45,719 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:24:51,753 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 14:24:51,760 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:24:55,182 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 14:24:55,191 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:24:56,973 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 14:24:59,871 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 14:25:06,918 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:26:12,778 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 14:26:12,786 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:26:16,233 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 14:26:16,240 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:26:17,989 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 14:26:21,014 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 14:26:28,354 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:28:56,840 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 14:28:56,847 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:29:00,520 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 14:29:00,526 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:29:02,290 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 14:29:05,169 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 14:29:31,713 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 14:29:31,719 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:29:35,317 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 14:29:35,324 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:29:37,030 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 14:29:39,930 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 14:29:46,384 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:30:33,692 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-67) Start refreshing all users data >2013-03-07 14:30:34,535 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2013-03-07 14:30:34,538 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) Attempting to update VM OVFs in Data Center Default >2013-03-07 14:30:34,539 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) Successfully updated VM OVFs in Data Center Default >2013-03-07 14:30:34,539 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) Attempting to update template OVFs in Data Center Default >2013-03-07 14:30:34,539 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) Succesfully updated templates OVFs in Data Center Default >2013-03-07 14:30:34,540 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-07 14:30:34,540 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-07 14:32:12,622 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 14:32:12,630 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:32:16,227 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 14:32:16,233 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:32:17,927 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 14:32:20,829 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 14:32:27,147 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:33:15,029 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 14:33:15,036 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:33:18,796 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 14:33:18,804 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:33:20,518 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 14:33:23,367 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 14:33:29,933 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:35:21,728 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 14:35:21,736 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:35:25,253 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 14:35:25,260 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:35:26,979 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 14:35:29,849 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 14:35:36,236 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:42:06,522 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 14:42:06,529 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:42:10,083 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 14:42:10,090 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:42:11,778 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 14:42:14,629 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-07 14:42:21,067 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:46:23,130 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginAdminUserCommand internal: false. >2013-03-07 14:46:23,182 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 14:46:23,192 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:46:23,193 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:46:24,994 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (WebAdminUpperRightButtonURL) with null version, using default general for version >2013-03-07 14:47:23,266 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 14:47:23,275 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:47:23,275 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:47:37,128 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-3-thread-48) [7444fe77] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-07 14:47:37,131 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-48) [7444fe77] START, SetVdsStatusVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 2f7efde4 >2013-03-07 14:47:37,135 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-48) [7444fe77] FINISH, SetVdsStatusVDSCommand, log id: 2f7efde4 >2013-03-07 14:47:37,186 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-23) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 : cheetah01.scl.lab.tlv.redhat.com >2013-03-07 14:47:37,198 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) Clearing cache of pool: 5849b030-626e-47cb-ad90-3ce782d831b3 for problematic entities of VDS: cheetah01.scl.lab.tlv.redhat.com. >2013-03-07 14:47:37,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (QuartzScheduler_Worker-23) START, DisconnectStoragePoolVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 1), log id: 3ce6f2f0 >2013-03-07 14:47:37,396 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-3-thread-48) [7444fe77] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-07 14:47:40,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (QuartzScheduler_Worker-23) FINISH, DisconnectStoragePoolVDSCommand, log id: 3ce6f2f0 >2013-03-07 14:47:40,230 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (QuartzScheduler_Worker-23) [e983899] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 14:47:40,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (QuartzScheduler_Worker-23) [e983899] START, DisconnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, storageType = NFS, connectionList = [{ id: 4e470cfe-b114-4fb0-8f52-fc538d2b740b, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7dcefcda >2013-03-07 14:47:44,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (QuartzScheduler_Worker-23) [e983899] FINISH, DisconnectStorageServerVDSCommand, return: {4e470cfe-b114-4fb0-8f52-fc538d2b740b=0}, log id: 7dcefcda >2013-03-07 14:48:16,413 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-7) [_WEBADMIN_14436771_Configure_Local_Storage] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-07 14:48:16,530 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-1) [_WEBADMIN_14436771_Configure_Local_Storage] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: af0d05f3-f358-4df5-ac81-35684017b815 Type: StoragePool >2013-03-07 14:48:16,632 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp-/127.0.0.1:8702-5) [_WEBADMIN_14436771_Configure_Local_Storage] Running command: ChangeVDSClusterCommand internal: false. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS, ID: 0522bd09-2a55-4b55-95c9-2c8fb81d9382 Type: VdsGroups >2013-03-07 14:48:16,751 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-5) [_WEBADMIN_14436771_Configure_Local_Storage] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-07 14:48:16,775 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-5) [_WEBADMIN_14436771_Configure_Local_Storage] Lock freed to object EngineLock [exclusiveLocks= key: af0d05f3-f358-4df5-ac81-35684017b815 value: REGISTER_VDS >, sharedLocks= ] >2013-03-07 14:48:16,826 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-10) [_WEBADMIN_14436771_Configure_Local_Storage] Lock Acquired to object EngineLock [exclusiveLocks= key: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 value: VDS >, sharedLocks= ] >2013-03-07 14:48:16,834 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-10) [_WEBADMIN_14436771_Configure_Local_Storage] Running command: ActivateVdsCommand internal: false. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-07 14:48:16,846 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-10) [_WEBADMIN_14436771_Configure_Local_Storage] START, SetVdsStatusVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, status=Unassigned, nonOperationalReason=NONE), log id: 34a737d2 >2013-03-07 14:48:16,854 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-10) [_WEBADMIN_14436771_Configure_Local_Storage] FINISH, SetVdsStatusVDSCommand, log id: 34a737d2 >2013-03-07 14:48:16,857 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [_WEBADMIN_14436771_Configure_Local_Storage] START, ActivateVdsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0), log id: 40bddf33 >2013-03-07 14:48:16,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (ajp-/127.0.0.1:8702-10) [_WEBADMIN_14436771_Configure_Local_Storage] START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: 1b4cc798 >2013-03-07 14:48:16,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (ajp-/127.0.0.1:8702-10) [_WEBADMIN_14436771_Configure_Local_Storage] FINISH, GetHardwareInfoVDSCommand, log id: 1b4cc798 >2013-03-07 14:48:16,949 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (ajp-/127.0.0.1:8702-10) [3baf04d0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-07 14:48:16,952 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (ajp-/127.0.0.1:8702-10) [e7d6f77] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-07 14:48:16,954 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [e7d6f77] FINISH, ActivateVdsVDSCommand, log id: 40bddf33 >2013-03-07 14:48:16,976 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-10) [e7d6f77] Lock freed to object EngineLock [exclusiveLocks= key: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 value: VDS >, sharedLocks= ] >2013-03-07 14:48:17,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: 280b2b61 >2013-03-07 14:48:17,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) FINISH, GetHardwareInfoVDSCommand, log id: 280b2b61 >2013-03-07 14:48:17,391 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-58) [7a900072] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 14:48:17,411 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-58) [654c28df] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-07 14:48:17,412 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-50) [7a900072] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2013-03-07 14:48:17,418 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-58) [3ca4e62b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-07 14:48:23,262 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 14:48:23,269 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:48:23,269 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:48:27,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-11) [_WEBADMIN_14436771_Configure_Local_Storage] START, ValidateStorageServerConnectionVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: null, connection: /data, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 644ec737 >2013-03-07 14:48:27,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-11) [_WEBADMIN_14436771_Configure_Local_Storage] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 644ec737 >2013-03-07 14:48:27,115 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-11) [_WEBADMIN_14436771_Configure_Local_Storage] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-07 14:48:27,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-11) [_WEBADMIN_14436771_Configure_Local_Storage] START, ConnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: null, connection: /data, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4ef6bedb >2013-03-07 14:48:27,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-11) [_WEBADMIN_14436771_Configure_Local_Storage] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 4ef6bedb >2013-03-07 14:48:27,273 INFO [org.ovirt.engine.core.bll.storage.AddLocalStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [_WEBADMIN_14436771_Configure_Local_Storage] Running command: AddLocalStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-07 14:48:27,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-6) [_WEBADMIN_14436771_Configure_Local_Storage] START, CreateStorageDomainVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storageDomain=org.ovirt.engine.core.common.businessentities.StorageDomainStatic@73c71891, args=/data), log id: 231c20 >2013-03-07 14:48:31,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-6) [_WEBADMIN_14436771_Configure_Local_Storage] FINISH, CreateStorageDomainVDSCommand, log id: 231c20 >2013-03-07 14:48:31,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-6) [_WEBADMIN_14436771_Configure_Local_Storage] START, GetStorageDomainStatsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storageDomainId=77c935fb-bd6e-4f85-84e0-2377bf89e0b8), log id: 3ddd01c3 >2013-03-07 14:48:31,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-6) [_WEBADMIN_14436771_Configure_Local_Storage] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage_domains@8eb48d05, log id: 3ddd01c3 >2013-03-07 14:48:31,481 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-6) [7b027258] Running command: AttachStorageDomainToPoolCommand internal: true. Entities affected : ID: 77c935fb-bd6e-4f85-84e0-2377bf89e0b8 Type: Storage >2013-03-07 14:48:31,487 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-6) [77a38664] Lock Acquired to object EngineLock [exclusiveLocks= key: af0d05f3-f358-4df5-ac81-35684017b815 value: POOL >, sharedLocks= ] >2013-03-07 14:48:31,496 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-6) [77a38664] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected : ID: af0d05f3-f358-4df5-ac81-35684017b815 Type: StoragePool >2013-03-07 14:48:31,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-6) [1de0fd15] START, ValidateStorageServerConnectionVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 1b2864b1-237c-4316-bff5-b7044c234a9d, connection: /data, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3d1a0587 >2013-03-07 14:48:31,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-6) [1de0fd15] FINISH, ValidateStorageServerConnectionVDSCommand, return: {1b2864b1-237c-4316-bff5-b7044c234a9d=0}, log id: 3d1a0587 >2013-03-07 14:48:31,640 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8702-6) [1de0fd15] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-07 14:48:31,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-6) [1de0fd15] START, ConnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 1b2864b1-237c-4316-bff5-b7044c234a9d, connection: /data, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7e1032ae >2013-03-07 14:48:31,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-6) [1de0fd15] FINISH, ConnectStorageServerVDSCommand, return: {1b2864b1-237c-4316-bff5-b7044c234a9d=0}, log id: 7e1032ae >2013-03-07 14:48:31,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-6) [1de0fd15] START, CreateStoragePoolVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId=af0d05f3-f358-4df5-ac81-35684017b815, storageType=LOCALFS, storagePoolName=cheetah01-scl-lab-tlv-redhat-com-Local, masterDomainId=77c935fb-bd6e-4f85-84e0-2377bf89e0b8, domainsIdList=[77c935fb-bd6e-4f85-84e0-2377bf89e0b8], masterVersion=1), log id: 779afae0 >2013-03-07 14:48:52,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-6) [1de0fd15] FINISH, CreateStoragePoolVDSCommand, log id: 779afae0 >2013-03-07 14:48:52,935 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-6) [1de0fd15] Lock freed to object EngineLock [exclusiveLocks= key: af0d05f3-f358-4df5-ac81-35684017b815 value: POOL >, sharedLocks= ] >2013-03-07 14:48:52,938 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [59614652] Lock Acquired to object EngineLock [exclusiveLocks= key: 77c935fb-bd6e-4f85-84e0-2377bf89e0b8 value: STORAGE >, sharedLocks= ] >2013-03-07 14:48:52,940 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [59614652] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: 77c935fb-bd6e-4f85-84e0-2377bf89e0b8 Type: Storage >2013-03-07 14:48:52,958 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [59614652] Lock freed to object EngineLock [exclusiveLocks= key: 77c935fb-bd6e-4f85-84e0-2377bf89e0b8 value: STORAGE >, sharedLocks= ] >2013-03-07 14:48:52,959 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [59614652] ActivateStorage Domain. Before Connect all hosts to pool. Time:3/7/13 2:48 PM >2013-03-07 14:48:52,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-3-thread-50) [6fccb60f] START, ValidateStorageServerConnectionVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 1b2864b1-237c-4316-bff5-b7044c234a9d, connection: /data, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 638e7826 >2013-03-07 14:48:52,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-3-thread-50) [6fccb60f] FINISH, ValidateStorageServerConnectionVDSCommand, return: {1b2864b1-237c-4316-bff5-b7044c234a9d=0}, log id: 638e7826 >2013-03-07 14:48:52,991 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-3-thread-50) [6fccb60f] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-07 14:48:52,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-3-thread-50) [6fccb60f] START, ConnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 1b2864b1-237c-4316-bff5-b7044c234a9d, connection: /data, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2494e383 >2013-03-07 14:48:53,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-3-thread-50) [6fccb60f] FINISH, ConnectStorageServerVDSCommand, return: {1b2864b1-237c-4316-bff5-b7044c234a9d=0}, log id: 2494e383 >2013-03-07 14:48:53,022 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-6) [59614652] START, ActivateStorageDomainVDSCommand( storagePoolId = af0d05f3-f358-4df5-ac81-35684017b815, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 77c935fb-bd6e-4f85-84e0-2377bf89e0b8), log id: 1daf7369 >2013-03-07 14:48:53,036 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (ajp-/127.0.0.1:8702-6) [6342ff58] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: af0d05f3-f358-4df5-ac81-35684017b815 Type: StoragePool >2013-03-07 14:48:53,096 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-6) [6342ff58] hostFromVds::selectedVds - cheetah01.scl.lab.tlv.redhat.com, spmStatus Unknown_Pool, storage pool cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 14:48:53,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-6) [6342ff58] START, ConnectStoragePoolVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = af0d05f3-f358-4df5-ac81-35684017b815, vds_spm_id = 1, masterDomainId = 77c935fb-bd6e-4f85-84e0-2377bf89e0b8, masterVersion = 1), log id: 309cf5a >2013-03-07 14:48:57,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-6) [6342ff58] FINISH, ConnectStoragePoolVDSCommand, log id: 309cf5a >2013-03-07 14:48:57,220 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-6) [6342ff58] hostFromVds::Connected host to pool - selectedVds - cheetah01.scl.lab.tlv.redhat.com, spmStatus Free, storage pool cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 14:48:57,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-6) [6342ff58] starting spm on vds cheetah01.scl.lab.tlv.redhat.com, storage pool cheetah01-scl-lab-tlv-redhat-com-Local, prevId -1, LVER -1 >2013-03-07 14:48:57,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-6) [6342ff58] START, SpmStartVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = af0d05f3-f358-4df5-ac81-35684017b815, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 496eed82 >2013-03-07 14:48:57,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-6) [6342ff58] spmStart polling started: taskId = 90dfa3cd-6c10-44c4-bb5a-da45a7f4fcbb >2013-03-07 14:49:18,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-6) [6342ff58] spmStart polling ended: taskId = 90dfa3cd-6c10-44c4-bb5a-da45a7f4fcbb task status = finished >2013-03-07 14:49:18,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-6) [6342ff58] spmStart polling ended, spm status: SPM >2013-03-07 14:49:18,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8702-6) [6342ff58] START, HSMClearTaskVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, taskId=90dfa3cd-6c10-44c4-bb5a-da45a7f4fcbb), log id: 47a68a1d >2013-03-07 14:49:18,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8702-6) [6342ff58] FINISH, HSMClearTaskVDSCommand, log id: 47a68a1d >2013-03-07 14:49:18,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-6) [6342ff58] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@71d54705, log id: 496eed82 >2013-03-07 14:49:18,804 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-6) [6342ff58] Initialize Irs proxy from vds: cheetah01.scl.lab.tlv.redhat.com >2013-03-07 14:49:18,843 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-6) [6342ff58] FINISH, ActivateStorageDomainVDSCommand, log id: 1daf7369 >2013-03-07 14:49:18,843 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [6342ff58] ActivateStorage Domain. After Connect all hosts to pool. Time:3/7/13 2:49 PM >2013-03-07 14:49:18,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) [6342ff58] START, ConnectStoragePoolVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = af0d05f3-f358-4df5-ac81-35684017b815, vds_spm_id = 1, masterDomainId = 77c935fb-bd6e-4f85-84e0-2377bf89e0b8, masterVersion = 1), log id: 16939d6a >2013-03-07 14:49:18,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) [6342ff58] FINISH, ConnectStoragePoolVDSCommand, log id: 16939d6a >2013-03-07 14:49:18,896 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [6342ff58] ActivateStorage Domain. After change storage pool status in vds. Time:3/7/13 2:49 PM >2013-03-07 14:49:23,265 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 14:49:23,272 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:49:23,272 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:50:23,269 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 14:50:23,276 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:50:23,277 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:51:23,289 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-07 14:51:23,295 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:51:23,296 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:52:23,269 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 14:52:23,278 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:52:23,278 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:53:23,277 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 14:53:23,286 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:53:23,286 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:54:23,273 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 14:54:23,280 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:54:23,280 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:55:23,274 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-07 14:55:23,284 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:55:23,284 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:56:23,276 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-07 14:56:23,292 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:56:23,293 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:57:23,278 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-07 14:57:23,298 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:57:23,298 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:58:23,286 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 14:58:23,295 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:58:23,296 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 14:59:23,281 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 14:59:23,288 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 14:59:23,288 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:00:23,283 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 15:00:23,290 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:00:23,290 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:01:23,284 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 15:01:23,290 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:01:23,291 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:02:23,287 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 15:02:23,297 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:02:23,297 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:03:23,287 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-07 15:03:23,294 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:03:23,295 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:03:39,007 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) domain 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126:D1 in problem. vds: cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:04:15,360 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-30) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-07 15:04:18,369 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-15) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-07 15:04:21,377 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-36) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-07 15:04:22,280 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-38) [27ea8a22] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 15:04:22,316 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-38) [27ea8a22] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2013-03-07 15:04:23,289 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 15:04:23,294 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:04:23,294 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:04:24,384 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-37) ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-07 15:04:27,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1dbc7410 >2013-03-07 15:04:27,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) FINISH, GetHardwareInfoVDSCommand, log id: 1dbc7410 >2013-03-07 15:04:27,563 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-46) [11d4fe6c] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 15:04:27,568 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-46) [1c8adb8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 15:04:27,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-46) [1c8adb8] START, ConnectStorageServerVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, storageType = NFS, connectionList = [{ id: 4e470cfe-b114-4fb0-8f52-fc538d2b740b, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: eaf80b8 >2013-03-07 15:04:27,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-46) [1c8adb8] FINISH, ConnectStorageServerVDSCommand, return: {4e470cfe-b114-4fb0-8f52-fc538d2b740b=0}, log id: eaf80b8 >2013-03-07 15:04:27,604 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-46) [1c8adb8] Host cheetah02.scl.lab.tlv.redhat.com storage connection was succeeded >2013-03-07 15:04:27,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 15097dcd >2013-03-07 15:05:23,290 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 15:05:23,300 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:05:23,301 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:05:35,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 15:05:35,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:05:35,484 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 15:05:35,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 15097dcd >2013-03-07 15:05:35,493 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [7b60177e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:05:35,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (pool-3-thread-48) [7b60177e] START, ResetIrsVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, vdsId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, ignoreStopFailed = true), log id: 74db8253 >2013-03-07 15:05:35,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-3-thread-48) [7b60177e] START, SpmStopVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3), log id: 24df13d5 >2013-03-07 15:05:35,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-3-thread-48) [7b60177e] SpmStopVDSCommand:: vds cheetah02.scl.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 15:05:35,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-3-thread-48) [7b60177e] FINISH, SpmStopVDSCommand, log id: 24df13d5 >2013-03-07 15:05:35,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (pool-3-thread-48) [7b60177e] FINISH, ResetIrsVDSCommand, log id: 74db8253 >2013-03-07 15:05:35,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [7b60177e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 487a654c >2013-03-07 15:05:35,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [7b60177e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 15:05:35,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [7b60177e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 487a654c >2013-03-07 15:05:35,530 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [1c8adb8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 15:05:35,546 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-46) [1ab21e09] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:05:35,547 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [1ab21e09] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 15d06c50 >2013-03-07 15:05:35,549 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [1ab21e09] FINISH, SetVdsStatusVDSCommand, log id: 15d06c50 >2013-03-07 15:05:35,557 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-46) [1ab21e09] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 15:05:35,580 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-46) [2e15c012] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:05:35,587 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-46) [111f6844] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:05:35,589 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-46) [111f6844] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 15:06:23,291 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 15:06:23,297 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:06:23,297 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:07:23,293 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-07 15:07:23,305 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:07:23,305 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:08:23,299 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 15:08:23,308 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:08:23,308 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:09:23,296 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 15:09:23,303 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:09:23,303 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) Autorecovering 1 hosts >2013-03-07 15:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:10:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [599f05f3] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:10:00,010 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [599f05f3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [599f05f3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6053da90 >2013-03-07 15:10:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [599f05f3] FINISH, SetVdsStatusVDSCommand, log id: 6053da90 >2013-03-07 15:10:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-24) [599f05f3] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 238ce23d >2013-03-07 15:10:00,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [599f05f3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3ddab7f5 >2013-03-07 15:10:00,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [599f05f3] FINISH, GetHardwareInfoVDSCommand, log id: 3ddab7f5 >2013-03-07 15:10:00,424 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-24) [519e01f5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:10:00,426 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-24) [70aabd47] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:10:00,428 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-24) [70aabd47] FINISH, ActivateVdsVDSCommand, log id: 238ce23d >2013-03-07 15:10:00,449 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [70aabd47] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:10:00,454 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [70aabd47] Autorecovering 1 storage domains >2013-03-07 15:10:00,454 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [70aabd47] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 15:10:00,455 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [7c6dabac] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:10:00,455 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [7c6dabac] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 3:10 PM >2013-03-07 15:10:00,459 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [7c6dabac] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 3:10 PM >2013-03-07 15:10:02,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6d3a6a4a >2013-03-07 15:10:02,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) FINISH, GetHardwareInfoVDSCommand, log id: 6d3a6a4a >2013-03-07 15:10:02,863 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-15) [147d7132] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 15:10:02,867 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-15) [35ccaa34] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 15:10:02,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6c9e1dd0 >2013-03-07 15:10:23,298 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 15:10:23,305 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:10:23,305 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:11:07,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 15:11:07,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:11:07,867 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 15:11:07,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 6c9e1dd0 >2013-03-07 15:11:07,872 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [d53964d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:11:07,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [d53964d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 704f8171 >2013-03-07 15:11:07,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [d53964d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 15:11:07,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [d53964d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 704f8171 >2013-03-07 15:11:07,899 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [d53964d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 15:11:07,903 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [35ccaa34] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 15:11:07,912 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-15) [30d6daee] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:11:07,914 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [30d6daee] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: c51e89 >2013-03-07 15:11:07,915 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [30d6daee] FINISH, SetVdsStatusVDSCommand, log id: c51e89 >2013-03-07 15:11:07,934 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-15) [30d6daee] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 15:11:07,955 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-15) [3ca7ef03] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:11:07,958 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-15) [35751517] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:11:07,959 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-15) [35751517] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 15:11:23,300 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 15:11:23,306 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:11:23,307 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:12:23,310 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-07 15:12:23,318 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:12:23,319 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:13:23,303 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 15:13:23,310 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:13:23,310 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:14:23,304 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 15:14:23,311 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:14:23,312 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:15:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) Autorecovering 1 hosts >2013-03-07 15:15:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:15:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [10ead8c8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:15:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [10ead8c8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [10ead8c8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2d1b33a5 >2013-03-07 15:15:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [10ead8c8] FINISH, SetVdsStatusVDSCommand, log id: 2d1b33a5 >2013-03-07 15:15:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-77) [10ead8c8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7ca953d6 >2013-03-07 15:15:00,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [10ead8c8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 41e0b70b >2013-03-07 15:15:00,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [10ead8c8] FINISH, GetHardwareInfoVDSCommand, log id: 41e0b70b >2013-03-07 15:15:00,096 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-77) [4075b5d1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:15:00,099 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-77) [1837497e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:15:00,100 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-77) [1837497e] FINISH, ActivateVdsVDSCommand, log id: 7ca953d6 >2013-03-07 15:15:00,121 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [1837497e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:15:00,122 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-77) [1837497e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 15:15:00,124 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [1837497e] Autorecovering 1 storage domains >2013-03-07 15:15:00,124 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [1837497e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 15:15:00,125 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [5f9d4414] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:15:00,125 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [5f9d4414] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 3:15 PM >2013-03-07 15:15:00,131 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [5f9d4414] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 3:15 PM >2013-03-07 15:15:02,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 61810025 >2013-03-07 15:15:02,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) FINISH, GetHardwareInfoVDSCommand, log id: 61810025 >2013-03-07 15:15:02,224 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-72) [2a6706a3] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 15:15:02,228 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-72) [41f541f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 15:15:02,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3c4dacfb >2013-03-07 15:15:23,306 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-07 15:15:23,313 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:15:23,314 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:16:06,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 15:16:06,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:16:06,211 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 15:16:06,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 3c4dacfb >2013-03-07 15:16:06,215 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [1e088e26] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:16:06,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1e088e26] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 43c0c4ca >2013-03-07 15:16:06,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [1e088e26] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 15:16:06,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1e088e26] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 43c0c4ca >2013-03-07 15:16:06,244 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [1e088e26] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 15:16:06,244 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [41f541f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 15:16:06,257 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-72) [2921e15e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:16:06,259 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [2921e15e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3e1a65b2 >2013-03-07 15:16:06,261 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [2921e15e] FINISH, SetVdsStatusVDSCommand, log id: 3e1a65b2 >2013-03-07 15:16:06,268 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [2921e15e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 15:16:06,291 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [4fd01d51] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:16:06,293 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [30286d96] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:16:06,295 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-72) [30286d96] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 15:16:23,308 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-07 15:16:23,315 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:16:23,315 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:17:23,311 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 15:17:23,317 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:17:23,318 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:18:23,312 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 15:18:23,317 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:18:23,317 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:19:23,314 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-07 15:19:23,324 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:19:23,324 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:20:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) Autorecovering 1 hosts >2013-03-07 15:20:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [657dc083] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [657dc083] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [657dc083] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 469c2443 >2013-03-07 15:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [657dc083] FINISH, SetVdsStatusVDSCommand, log id: 469c2443 >2013-03-07 15:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-18) [657dc083] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 59266320 >2013-03-07 15:20:00,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [657dc083] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 506cc012 >2013-03-07 15:20:00,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [657dc083] FINISH, GetHardwareInfoVDSCommand, log id: 506cc012 >2013-03-07 15:20:00,104 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [6bdcef6a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:20:00,106 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [10bac024] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:20:00,108 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-18) [10bac024] FINISH, ActivateVdsVDSCommand, log id: 59266320 >2013-03-07 15:20:00,128 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [10bac024] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:20:00,129 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [10bac024] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 15:20:00,130 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [10bac024] Autorecovering 1 storage domains >2013-03-07 15:20:00,130 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [10bac024] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 15:20:00,131 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [3fa3ab4e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:20:00,132 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [3fa3ab4e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 3:20 PM >2013-03-07 15:20:00,135 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [3fa3ab4e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 3:20 PM >2013-03-07 15:20:00,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-23) [e983899] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1789eac9 >2013-03-07 15:20:00,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-23) [e983899] FINISH, GetHardwareInfoVDSCommand, log id: 1789eac9 >2013-03-07 15:20:00,541 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-23) [140dee4e] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 15:20:00,544 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-23) [1fe01c6b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 15:20:00,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 160c80aa >2013-03-07 15:20:23,315 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-07 15:20:23,328 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:20:23,328 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:21:06,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 15:21:06,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:21:06,597 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 15:21:06,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 160c80aa >2013-03-07 15:21:06,601 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [37ddcf45] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:21:06,628 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [37ddcf45] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 39812145 >2013-03-07 15:21:06,628 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [37ddcf45] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 15:21:06,628 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [37ddcf45] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 39812145 >2013-03-07 15:21:06,631 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [37ddcf45] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 15:21:06,635 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [1fe01c6b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 15:21:06,643 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-23) [2acd632] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:21:06,645 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-23) [2acd632] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 17199d9d >2013-03-07 15:21:06,646 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-23) [2acd632] FINISH, SetVdsStatusVDSCommand, log id: 17199d9d >2013-03-07 15:21:06,654 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-23) [2acd632] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 15:21:06,674 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-23) [2dde530d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:21:06,677 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-23) [7d35bcf3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:21:06,679 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-23) [7d35bcf3] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 15:21:23,321 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-07 15:21:23,328 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:21:23,328 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:22:23,317 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-07 15:22:23,325 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:22:23,325 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:23:23,321 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 15:23:23,328 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:23:23,328 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:24:23,321 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 15:24:23,328 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:24:23,328 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) Autorecovering 1 hosts >2013-03-07 15:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [3aa862f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [3aa862f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [3aa862f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 41b0cfbf >2013-03-07 15:25:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [3aa862f] FINISH, SetVdsStatusVDSCommand, log id: 41b0cfbf >2013-03-07 15:25:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-68) [3aa862f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 306a8542 >2013-03-07 15:25:00,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [3aa862f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4fe9d0ee >2013-03-07 15:25:00,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [3aa862f] FINISH, GetHardwareInfoVDSCommand, log id: 4fe9d0ee >2013-03-07 15:25:00,584 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-68) [66cf5427] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:25:00,587 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-68) [5d451250] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:25:00,589 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-68) [5d451250] FINISH, ActivateVdsVDSCommand, log id: 306a8542 >2013-03-07 15:25:00,610 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [5d451250] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:25:00,611 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-68) [5d451250] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 15:25:00,612 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [5d451250] Autorecovering 1 storage domains >2013-03-07 15:25:00,612 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [5d451250] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 15:25:00,613 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [65873a9c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:25:00,613 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [65873a9c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 3:25 PM >2013-03-07 15:25:00,616 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [65873a9c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 3:25 PM >2013-03-07 15:25:00,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 70d71270 >2013-03-07 15:25:00,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) FINISH, GetHardwareInfoVDSCommand, log id: 70d71270 >2013-03-07 15:25:00,940 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-75) [28363b6a] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 15:25:00,945 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-75) [41428c3f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 15:25:00,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 29d08aa >2013-03-07 15:25:23,324 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 15:25:23,331 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:25:23,331 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:26:05,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 15:26:05,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:26:05,954 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 15:26:05,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 29d08aa >2013-03-07 15:26:05,957 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [1018ed3c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:26:05,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [1018ed3c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 49f3e2b >2013-03-07 15:26:05,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [1018ed3c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 15:26:05,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [1018ed3c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 49f3e2b >2013-03-07 15:26:05,977 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [1018ed3c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 15:26:05,978 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [41428c3f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 15:26:05,988 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-75) [384ad6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:26:05,990 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [384ad6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4d9f7a59 >2013-03-07 15:26:05,991 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [384ad6] FINISH, SetVdsStatusVDSCommand, log id: 4d9f7a59 >2013-03-07 15:26:05,997 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-75) [384ad6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 15:26:06,020 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [48799970] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:26:06,022 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [ec4766d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:26:06,024 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-75) [ec4766d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 15:26:23,325 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 15:26:23,330 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:26:23,330 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:27:23,326 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 15:27:23,333 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:27:23,333 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:28:23,328 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 15:28:23,335 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:28:23,335 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:29:23,329 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 15:29:23,336 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:29:23,336 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) Autorecovering 1 hosts >2013-03-07 15:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [65d3413c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [65d3413c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [65d3413c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 13e7e28c >2013-03-07 15:30:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [65d3413c] FINISH, SetVdsStatusVDSCommand, log id: 13e7e28c >2013-03-07 15:30:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [65d3413c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6fa33362 >2013-03-07 15:30:00,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [65d3413c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7ef349cb >2013-03-07 15:30:00,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [65d3413c] FINISH, GetHardwareInfoVDSCommand, log id: 7ef349cb >2013-03-07 15:30:00,095 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-14) [12937717] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:30:00,098 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-14) [41bdf0e9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:30:00,099 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [41bdf0e9] FINISH, ActivateVdsVDSCommand, log id: 6fa33362 >2013-03-07 15:30:00,119 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [41bdf0e9] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:30:00,120 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-14) [41bdf0e9] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 15:30:00,121 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [41bdf0e9] Autorecovering 1 storage domains >2013-03-07 15:30:00,121 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [41bdf0e9] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 15:30:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [1cae830a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:30:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [1cae830a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 3:30 PM >2013-03-07 15:30:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [1cae830a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 3:30 PM >2013-03-07 15:30:00,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: d231a91 >2013-03-07 15:30:00,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) FINISH, GetHardwareInfoVDSCommand, log id: d231a91 >2013-03-07 15:30:00,272 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-16) [218d2db4] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 15:30:00,276 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-16) [57e3989b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 15:30:00,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 39ad7e8f >2013-03-07 15:30:23,331 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 15:30:23,342 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:30:23,342 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:30:33,694 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-48) Start refreshing all users data >2013-03-07 15:30:34,541 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2013-03-07 15:30:34,543 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 15:30:34,544 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 15:30:34,544 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 15:30:34,545 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 15:30:34,545 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 15:30:34,546 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 15:31:04,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 15:31:04,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:31:04,256 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 15:31:04,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 39ad7e8f >2013-03-07 15:31:04,260 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [78325653] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:31:04,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [78325653] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 350b7162 >2013-03-07 15:31:04,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [78325653] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 15:31:04,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [78325653] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 350b7162 >2013-03-07 15:31:04,282 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [78325653] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 15:31:04,282 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [57e3989b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 15:31:04,294 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-16) [3eafd77f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:31:04,295 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [3eafd77f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 67bfd752 >2013-03-07 15:31:04,296 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [3eafd77f] FINISH, SetVdsStatusVDSCommand, log id: 67bfd752 >2013-03-07 15:31:04,303 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-16) [3eafd77f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 15:31:04,322 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-16) [230d209f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:31:04,326 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-16) [598a4655] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:31:04,327 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-16) [598a4655] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 15:31:23,333 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 15:31:23,340 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:31:23,340 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:32:23,335 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 15:32:23,342 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:32:23,343 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:33:23,337 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 15:33:23,344 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:33:23,344 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:34:23,338 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-07 15:34:23,346 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:34:23,346 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) Autorecovering 1 hosts >2013-03-07 15:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-62) [2b038f4d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-62) [2b038f4d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-62) [2b038f4d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1743c664 >2013-03-07 15:35:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-62) [2b038f4d] FINISH, SetVdsStatusVDSCommand, log id: 1743c664 >2013-03-07 15:35:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-62) [2b038f4d] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 19d2482c >2013-03-07 15:35:00,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-62) [2b038f4d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7e71bf6d >2013-03-07 15:35:00,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-62) [2b038f4d] FINISH, GetHardwareInfoVDSCommand, log id: 7e71bf6d >2013-03-07 15:35:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-62) [5c0fb01] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:35:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-62) [769342eb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:35:00,203 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-62) [769342eb] FINISH, ActivateVdsVDSCommand, log id: 19d2482c >2013-03-07 15:35:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-62) [769342eb] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:35:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-62) [769342eb] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 15:35:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) [769342eb] Autorecovering 1 storage domains >2013-03-07 15:35:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) [769342eb] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 15:35:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-62) [25797610] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:35:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-62) [25797610] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 3:35 PM >2013-03-07 15:35:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-62) [25797610] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 3:35 PM >2013-03-07 15:35:01,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 56d94333 >2013-03-07 15:35:01,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) FINISH, GetHardwareInfoVDSCommand, log id: 56d94333 >2013-03-07 15:35:01,572 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-70) [39cdae1f] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 15:35:01,576 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-70) [42be5c9d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 15:35:01,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6d9f9c04 >2013-03-07 15:35:23,342 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-07 15:35:23,350 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:35:23,350 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:36:06,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 15:36:06,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:36:06,550 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 15:36:06,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 6d9f9c04 >2013-03-07 15:36:06,554 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [434f971e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:36:06,574 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [434f971e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 28ec1e88 >2013-03-07 15:36:06,574 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [434f971e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 15:36:06,574 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [434f971e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 28ec1e88 >2013-03-07 15:36:06,577 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [434f971e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 15:36:06,577 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [42be5c9d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 15:36:06,589 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-70) [34a0e388] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:36:06,590 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [34a0e388] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 138194d4 >2013-03-07 15:36:06,592 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [34a0e388] FINISH, SetVdsStatusVDSCommand, log id: 138194d4 >2013-03-07 15:36:06,599 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-70) [34a0e388] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 15:36:06,620 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-70) [5b47f59e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:36:06,623 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-70) [9a140e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:36:06,624 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-70) [9a140e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 15:36:23,344 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-07 15:36:23,349 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:36:23,349 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:37:23,345 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 15:37:23,353 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:37:23,355 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:38:23,347 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-07 15:38:23,352 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:38:23,353 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:39:23,349 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 15:39:23,358 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:39:23,359 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) Autorecovering 1 hosts >2013-03-07 15:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [2ec79241] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [2ec79241] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [2ec79241] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: e68aeea >2013-03-07 15:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [2ec79241] FINISH, SetVdsStatusVDSCommand, log id: e68aeea >2013-03-07 15:40:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-13) [2ec79241] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 53a0ae55 >2013-03-07 15:40:00,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [2ec79241] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6292dcd4 >2013-03-07 15:40:00,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [2ec79241] FINISH, GetHardwareInfoVDSCommand, log id: 6292dcd4 >2013-03-07 15:40:00,188 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-13) [6158e1d5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:40:00,191 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-13) [56a9ec9d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:40:00,192 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-13) [56a9ec9d] FINISH, ActivateVdsVDSCommand, log id: 53a0ae55 >2013-03-07 15:40:00,212 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [56a9ec9d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:40:00,213 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-13) [56a9ec9d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 15:40:00,215 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [56a9ec9d] Autorecovering 1 storage domains >2013-03-07 15:40:00,215 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [56a9ec9d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 15:40:00,215 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [1ca1e668] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:40:00,216 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [1ca1e668] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 3:40 PM >2013-03-07 15:40:00,219 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [1ca1e668] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 3:40 PM >2013-03-07 15:40:00,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [1cae830a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4058095c >2013-03-07 15:40:00,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [1cae830a] FINISH, GetHardwareInfoVDSCommand, log id: 4058095c >2013-03-07 15:40:00,870 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-14) [2ee4d3ab] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 15:40:00,874 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-14) [4f6f6f2e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 15:40:00,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: b2e9bc7 >2013-03-07 15:40:23,347 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 15:40:23,355 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:40:23,355 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:41:04,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 15:41:04,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:41:04,852 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 15:41:04,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: b2e9bc7 >2013-03-07 15:41:04,856 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [69cf241c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:41:04,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [69cf241c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 145802f9 >2013-03-07 15:41:04,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [69cf241c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 15:41:04,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [69cf241c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 145802f9 >2013-03-07 15:41:04,884 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [69cf241c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 15:41:04,885 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [4f6f6f2e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 15:41:04,898 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-14) [2438015b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:41:04,900 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [2438015b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3474a6e8 >2013-03-07 15:41:04,901 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [2438015b] FINISH, SetVdsStatusVDSCommand, log id: 3474a6e8 >2013-03-07 15:41:04,909 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-14) [2438015b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 15:41:04,934 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-14) [5ed4005d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:41:04,937 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-14) [2c941578] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:41:04,938 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-14) [2c941578] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 15:41:23,351 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 15:41:23,360 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:41:23,360 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:42:23,353 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 15:42:23,360 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:42:23,360 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:43:23,355 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 15:43:23,365 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:43:23,365 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:44:23,361 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-07 15:44:23,367 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:44:23,368 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) Autorecovering 1 hosts >2013-03-07 15:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [56c5a7d1] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [56c5a7d1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [56c5a7d1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 44417d91 >2013-03-07 15:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [56c5a7d1] FINISH, SetVdsStatusVDSCommand, log id: 44417d91 >2013-03-07 15:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [56c5a7d1] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2468b04e >2013-03-07 15:45:00,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [56c5a7d1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 35f5a6dc >2013-03-07 15:45:00,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [56c5a7d1] FINISH, GetHardwareInfoVDSCommand, log id: 35f5a6dc >2013-03-07 15:45:00,405 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-63) [788c3e86] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:45:00,419 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-63) [737cce8f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:45:00,420 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [737cce8f] FINISH, ActivateVdsVDSCommand, log id: 2468b04e >2013-03-07 15:45:00,441 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [737cce8f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:45:00,442 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-63) [737cce8f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 15:45:00,444 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [737cce8f] Autorecovering 1 storage domains >2013-03-07 15:45:00,444 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [737cce8f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 15:45:00,445 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [5f60280d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:45:00,445 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [5f60280d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 3:45 PM >2013-03-07 15:45:00,451 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [5f60280d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 3:45 PM >2013-03-07 15:45:02,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [3ca4e62b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6ca4babd >2013-03-07 15:45:02,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [3ca4e62b] FINISH, GetHardwareInfoVDSCommand, log id: 6ca4babd >2013-03-07 15:45:02,183 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-58) [5da57e29] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 15:45:02,186 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-58) [4416205c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 15:45:02,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 23c0915b >2013-03-07 15:45:23,359 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-07 15:45:23,367 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:45:23,367 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:46:05,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 15:46:05,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:46:05,157 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 15:46:05,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 23c0915b >2013-03-07 15:46:05,161 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [3ae17ca2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:46:05,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [3ae17ca2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 109afd53 >2013-03-07 15:46:05,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [3ae17ca2] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 15:46:05,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [3ae17ca2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 109afd53 >2013-03-07 15:46:05,180 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [3ae17ca2] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 15:46:05,180 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [4416205c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 15:46:05,193 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-58) [4abd704] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:46:05,195 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [4abd704] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5c97ac8 >2013-03-07 15:46:05,196 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [4abd704] FINISH, SetVdsStatusVDSCommand, log id: 5c97ac8 >2013-03-07 15:46:05,203 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-58) [4abd704] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 15:46:05,222 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-58) [73607967] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:46:05,224 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-58) [236f4673] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:46:05,229 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-58) [236f4673] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 15:46:23,360 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-07 15:46:23,367 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:46:23,367 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:47:23,362 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-07 15:47:23,369 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:47:23,370 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:48:23,363 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-07 15:48:23,370 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:48:23,370 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:49:23,365 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 15:49:23,372 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:49:23,372 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [1c44d063] Autorecovering 1 hosts >2013-03-07 15:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [1c44d063] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-7) [5177297a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-7) [5177297a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [5177297a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 91a5e71 >2013-03-07 15:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [5177297a] FINISH, SetVdsStatusVDSCommand, log id: 91a5e71 >2013-03-07 15:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-7) [5177297a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 68e9ddef >2013-03-07 15:50:00,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [5177297a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 617421e6 >2013-03-07 15:50:00,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [5177297a] FINISH, GetHardwareInfoVDSCommand, log id: 617421e6 >2013-03-07 15:50:00,099 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-7) [4fa92931] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:50:00,102 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-7) [69333679] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:50:00,104 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-7) [69333679] FINISH, ActivateVdsVDSCommand, log id: 68e9ddef >2013-03-07 15:50:00,127 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-7) [69333679] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:50:00,128 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-7) [69333679] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 15:50:00,129 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [69333679] Autorecovering 1 storage domains >2013-03-07 15:50:00,129 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [69333679] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 15:50:00,130 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-7) [56ef369] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:50:00,130 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-7) [56ef369] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 3:50 PM >2013-03-07 15:50:00,133 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-7) [56ef369] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 3:50 PM >2013-03-07 15:50:02,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [60232fff] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6a3e26de >2013-03-07 15:50:02,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [60232fff] FINISH, GetHardwareInfoVDSCommand, log id: 6a3e26de >2013-03-07 15:50:02,487 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-2) [1c766cd0] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 15:50:02,491 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-2) [382d5522] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 15:50:02,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1e90b7b2 >2013-03-07 15:50:23,368 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 15:50:23,377 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:50:23,377 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:51:07,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 15:51:07,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:51:07,527 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 15:51:07,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 1e90b7b2 >2013-03-07 15:51:07,531 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [1befef61] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:51:07,563 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1befef61] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 34acef03 >2013-03-07 15:51:07,564 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [1befef61] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 15:51:07,564 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1befef61] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 34acef03 >2013-03-07 15:51:07,566 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [1befef61] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 15:51:07,566 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [382d5522] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 15:51:07,577 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-2) [d9de7a7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:51:07,579 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [d9de7a7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 69413efd >2013-03-07 15:51:07,580 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [d9de7a7] FINISH, SetVdsStatusVDSCommand, log id: 69413efd >2013-03-07 15:51:07,587 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [d9de7a7] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 15:51:07,606 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [753fd877] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:51:07,608 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [769d0032] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:51:07,610 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-2) [769d0032] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 15:51:23,370 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 15:51:23,377 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:51:23,377 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:52:23,369 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 15:52:23,376 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:52:23,376 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:53:23,371 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-07 15:53:23,377 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:53:23,377 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:54:23,373 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-07 15:54:23,380 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:54:23,380 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) Autorecovering 1 hosts >2013-03-07 15:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [5cec1267] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [5cec1267] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [5cec1267] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5061028f >2013-03-07 15:55:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [5cec1267] FINISH, SetVdsStatusVDSCommand, log id: 5061028f >2013-03-07 15:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [5cec1267] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 468b1414 >2013-03-07 15:55:00,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [5cec1267] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 298a466a >2013-03-07 15:55:00,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [5cec1267] FINISH, GetHardwareInfoVDSCommand, log id: 298a466a >2013-03-07 15:55:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [4a6dcb5f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:55:00,263 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [1aa49cc9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:55:00,265 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [1aa49cc9] FINISH, ActivateVdsVDSCommand, log id: 468b1414 >2013-03-07 15:55:00,285 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [1aa49cc9] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 15:55:00,286 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [1aa49cc9] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 15:55:00,287 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [1aa49cc9] Autorecovering 1 storage domains >2013-03-07 15:55:00,287 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [1aa49cc9] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 15:55:00,288 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [6761b250] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:55:00,288 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [6761b250] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 3:55 PM >2013-03-07 15:55:00,291 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [6761b250] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 3:55 PM >2013-03-07 15:55:01,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 63767086 >2013-03-07 15:55:01,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) FINISH, GetHardwareInfoVDSCommand, log id: 63767086 >2013-03-07 15:55:01,861 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-59) [2b452ab3] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 15:55:01,865 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-59) [2ebf90ad] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 15:55:01,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 41a6ea08 >2013-03-07 15:55:23,375 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-07 15:55:23,383 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:55:23,383 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:56:06,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 15:56:06,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 15:56:06,851 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 15:56:06,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 41a6ea08 >2013-03-07 15:56:06,855 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [44b31d58] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 15:56:06,875 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [44b31d58] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 765ce0fa >2013-03-07 15:56:06,876 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [44b31d58] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 15:56:06,876 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [44b31d58] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 765ce0fa >2013-03-07 15:56:06,878 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [44b31d58] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 15:56:06,879 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [2ebf90ad] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 15:56:06,890 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-59) [1cfc7cf7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:56:06,891 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [1cfc7cf7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 77cb2934 >2013-03-07 15:56:06,893 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [1cfc7cf7] FINISH, SetVdsStatusVDSCommand, log id: 77cb2934 >2013-03-07 15:56:06,899 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [1cfc7cf7] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 15:56:06,924 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [7a9bb3fa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:56:06,927 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [42fdb2dc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 15:56:06,928 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-59) [42fdb2dc] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 15:56:23,376 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-07 15:56:23,383 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:56:23,383 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:57:23,378 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-07 15:57:23,385 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:57:23,385 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:58:23,380 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 15:58:23,387 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:58:23,387 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 15:59:23,386 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-07 15:59:23,393 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 15:59:23,393 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) Autorecovering 1 hosts >2013-03-07 16:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-99) [49d63c82] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-99) [49d63c82] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [49d63c82] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3c22e167 >2013-03-07 16:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [49d63c82] FINISH, SetVdsStatusVDSCommand, log id: 3c22e167 >2013-03-07 16:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-99) [49d63c82] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 406f5062 >2013-03-07 16:00:00,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [49d63c82] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 46fdc382 >2013-03-07 16:00:00,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [49d63c82] FINISH, GetHardwareInfoVDSCommand, log id: 46fdc382 >2013-03-07 16:00:00,720 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-99) [ce3a042] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:00:00,727 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-99) [1c3d7050] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:00:00,728 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-99) [1c3d7050] FINISH, ActivateVdsVDSCommand, log id: 406f5062 >2013-03-07 16:00:00,748 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-99) [1c3d7050] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:00:00,749 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-99) [1c3d7050] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 16:00:00,750 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [1c3d7050] Autorecovering 1 storage domains >2013-03-07 16:00:00,750 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [1c3d7050] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 16:00:00,751 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-99) [2beee732] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:00:00,751 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-99) [2beee732] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 4:00 PM >2013-03-07 16:00:00,755 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-99) [2beee732] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 4:00 PM >2013-03-07 16:00:01,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2f5e1343 >2013-03-07 16:00:01,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) FINISH, GetHardwareInfoVDSCommand, log id: 2f5e1343 >2013-03-07 16:00:01,174 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-5) [45b1135] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 16:00:01,178 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-5) [2bd1351c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 16:00:01,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6c6a29b8 >2013-03-07 16:00:23,382 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-07 16:00:23,390 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:00:23,390 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:01:06,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 16:01:06,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:01:06,166 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 16:01:06,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 6c6a29b8 >2013-03-07 16:01:06,170 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [3be085d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:01:06,186 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [3be085d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 49dceb4e >2013-03-07 16:01:06,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [3be085d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 16:01:06,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [3be085d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 49dceb4e >2013-03-07 16:01:06,189 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [3be085d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 16:01:06,190 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [2bd1351c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 16:01:06,201 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-5) [71691383] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:01:06,202 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [71691383] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 69688bb3 >2013-03-07 16:01:06,206 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [71691383] FINISH, SetVdsStatusVDSCommand, log id: 69688bb3 >2013-03-07 16:01:06,212 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-5) [71691383] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 16:01:06,231 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-5) [3ce3d299] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:01:06,234 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-5) [253426d6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:01:06,235 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-5) [253426d6] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 16:01:23,384 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-07 16:01:23,400 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:01:23,400 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:02:23,386 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 16:02:23,393 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:02:23,393 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:03:23,388 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 16:03:23,394 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:03:23,394 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:04:23,390 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 16:04:23,396 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:04:23,396 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) Autorecovering 1 hosts >2013-03-07 16:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-48) [7a6b37a8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-48) [7a6b37a8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [7a6b37a8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 58aa177 >2013-03-07 16:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [7a6b37a8] FINISH, SetVdsStatusVDSCommand, log id: 58aa177 >2013-03-07 16:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-48) [7a6b37a8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 465dcbf0 >2013-03-07 16:05:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [7a6b37a8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7f4f3d3b >2013-03-07 16:05:00,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [7a6b37a8] FINISH, GetHardwareInfoVDSCommand, log id: 7f4f3d3b >2013-03-07 16:05:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-48) [214b25e2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:05:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-48) [6e97c63d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:05:00,095 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-48) [6e97c63d] FINISH, ActivateVdsVDSCommand, log id: 465dcbf0 >2013-03-07 16:05:00,115 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-48) [6e97c63d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:05:00,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-48) [6e97c63d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 16:05:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [6e97c63d] Autorecovering 1 storage domains >2013-03-07 16:05:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [6e97c63d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 16:05:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-48) [28e66ea6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:05:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-48) [28e66ea6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 4:05 PM >2013-03-07 16:05:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-48) [28e66ea6] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 4:05 PM >2013-03-07 16:05:00,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 515d08de >2013-03-07 16:05:00,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) FINISH, GetHardwareInfoVDSCommand, log id: 515d08de >2013-03-07 16:05:00,483 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-52) [66dda95f] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 16:05:00,487 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-52) [26a319a4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 16:05:00,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5bd6eba0 >2013-03-07 16:05:23,419 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 16:05:23,444 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:05:23,444 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:06:04,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 16:06:04,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:06:04,789 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 16:06:04,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 5bd6eba0 >2013-03-07 16:06:04,793 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [45602c29] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:06:04,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [45602c29] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2f5155c0 >2013-03-07 16:06:04,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [45602c29] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 16:06:04,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [45602c29] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2f5155c0 >2013-03-07 16:06:04,814 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [26a319a4] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 16:06:04,825 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-52) [644d0d5f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:06:04,827 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [644d0d5f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7d1ccdb8 >2013-03-07 16:06:04,828 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [644d0d5f] FINISH, SetVdsStatusVDSCommand, log id: 7d1ccdb8 >2013-03-07 16:06:04,834 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-52) [644d0d5f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 16:06:04,855 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [592d534d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:06:04,858 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [35d5810] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:06:04,859 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-52) [35d5810] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 16:06:23,394 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 16:06:23,400 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:06:23,400 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:07:23,416 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 16:07:23,421 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:07:23,421 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:08:23,397 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 16:08:23,404 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:08:23,404 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:09:23,399 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-07 16:09:23,410 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:09:23,410 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) Autorecovering 1 hosts >2013-03-07 16:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [26fcd3fe] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [26fcd3fe] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [26fcd3fe] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7bfbb6a >2013-03-07 16:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [26fcd3fe] FINISH, SetVdsStatusVDSCommand, log id: 7bfbb6a >2013-03-07 16:10:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [26fcd3fe] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2d6d88ea >2013-03-07 16:10:00,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [26fcd3fe] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6be65d9d >2013-03-07 16:10:00,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [26fcd3fe] FINISH, GetHardwareInfoVDSCommand, log id: 6be65d9d >2013-03-07 16:10:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-96) [5ef9be1c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:10:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-96) [41d6bc45] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:10:00,205 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [41d6bc45] FINISH, ActivateVdsVDSCommand, log id: 2d6d88ea >2013-03-07 16:10:00,227 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [41d6bc45] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:10:00,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-96) [41d6bc45] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 16:10:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [41d6bc45] Autorecovering 1 storage domains >2013-03-07 16:10:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [41d6bc45] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 16:10:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [2311d9c2] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:10:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [2311d9c2] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 4:10 PM >2013-03-07 16:10:00,238 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [2311d9c2] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 4:10 PM >2013-03-07 16:10:02,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2aabae9c >2013-03-07 16:10:02,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) FINISH, GetHardwareInfoVDSCommand, log id: 2aabae9c >2013-03-07 16:10:02,111 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-94) [9c99a24] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 16:10:02,114 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-94) [6d8bd25d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 16:10:02,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 493c5c6 >2013-03-07 16:10:23,400 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-07 16:10:23,408 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:10:23,408 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:11:05,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 16:11:05,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:11:05,113 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 16:11:05,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 493c5c6 >2013-03-07 16:11:05,120 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [732b1a47] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:11:05,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [732b1a47] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6b37db15 >2013-03-07 16:11:05,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [732b1a47] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 16:11:05,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [732b1a47] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6b37db15 >2013-03-07 16:11:05,147 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [732b1a47] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 16:11:05,149 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [6d8bd25d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 16:11:05,158 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-94) [642b6842] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:11:05,160 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [642b6842] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 732e05bd >2013-03-07 16:11:05,161 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [642b6842] FINISH, SetVdsStatusVDSCommand, log id: 732e05bd >2013-03-07 16:11:05,167 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-94) [642b6842] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 16:11:05,193 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [7c14ae91] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:11:05,200 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [5a14534c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:11:05,201 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-94) [5a14534c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 16:11:23,401 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-07 16:11:23,408 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:11:23,409 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:12:23,405 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-07 16:12:23,412 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:12:23,412 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:13:23,406 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 16:13:23,412 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:13:23,412 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:14:23,408 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 16:14:23,415 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:14:23,415 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) Autorecovering 1 hosts >2013-03-07 16:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [16702981] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [16702981] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [16702981] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4a4f8ffb >2013-03-07 16:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [16702981] FINISH, SetVdsStatusVDSCommand, log id: 4a4f8ffb >2013-03-07 16:15:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-44) [16702981] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 585ba47e >2013-03-07 16:15:00,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [16702981] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6395a400 >2013-03-07 16:15:00,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [16702981] FINISH, GetHardwareInfoVDSCommand, log id: 6395a400 >2013-03-07 16:15:00,188 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-44) [46d3ce9c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:15:00,190 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-44) [39c95a85] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:15:00,192 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-44) [39c95a85] FINISH, ActivateVdsVDSCommand, log id: 585ba47e >2013-03-07 16:15:00,212 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [39c95a85] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:15:00,213 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-44) [39c95a85] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 16:15:00,214 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [39c95a85] Autorecovering 1 storage domains >2013-03-07 16:15:00,214 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [39c95a85] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 16:15:00,215 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [315006da] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:15:00,215 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [315006da] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 4:15 PM >2013-03-07 16:15:00,218 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [315006da] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 4:15 PM >2013-03-07 16:15:02,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 746bf789 >2013-03-07 16:15:02,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) FINISH, GetHardwareInfoVDSCommand, log id: 746bf789 >2013-03-07 16:15:02,445 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-41) [71b7a3fe] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 16:15:02,450 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-41) [2f92bb31] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 16:15:02,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 33aa5567 >2013-03-07 16:15:23,409 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 16:15:23,443 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:15:23,444 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:16:05,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 16:16:05,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:16:05,430 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 16:16:05,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 33aa5567 >2013-03-07 16:16:05,434 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [4a04f5d4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:16:05,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [4a04f5d4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6183634f >2013-03-07 16:16:05,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [4a04f5d4] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 16:16:05,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [4a04f5d4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6183634f >2013-03-07 16:16:05,456 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [4a04f5d4] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 16:16:05,458 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [2f92bb31] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 16:16:05,467 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-41) [2cb0a06e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:16:05,469 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [2cb0a06e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 510c5055 >2013-03-07 16:16:05,470 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [2cb0a06e] FINISH, SetVdsStatusVDSCommand, log id: 510c5055 >2013-03-07 16:16:05,475 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-41) [2cb0a06e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 16:16:05,494 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-41) [715df136] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:16:05,496 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-41) [262b6da6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:16:05,498 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-41) [262b6da6] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 16:16:23,410 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 16:16:23,417 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:16:23,417 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:17:23,411 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 16:17:23,418 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:17:23,418 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:18:23,413 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-07 16:18:23,419 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:18:23,419 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:19:23,415 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-07 16:19:23,422 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:19:23,422 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:20:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) Autorecovering 1 hosts >2013-03-07 16:20:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-86) [52a91cd] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-86) [52a91cd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [52a91cd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3fc6d235 >2013-03-07 16:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [52a91cd] FINISH, SetVdsStatusVDSCommand, log id: 3fc6d235 >2013-03-07 16:20:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-86) [52a91cd] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 243d1a5e >2013-03-07 16:20:00,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [52a91cd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1b8fcd1b >2013-03-07 16:20:00,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [52a91cd] FINISH, GetHardwareInfoVDSCommand, log id: 1b8fcd1b >2013-03-07 16:20:00,209 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-86) [63a6671c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:20:00,212 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-86) [7d827a46] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:20:00,214 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-86) [7d827a46] FINISH, ActivateVdsVDSCommand, log id: 243d1a5e >2013-03-07 16:20:00,234 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-86) [7d827a46] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:20:00,234 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-86) [7d827a46] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 16:20:00,236 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [7d827a46] Autorecovering 1 storage domains >2013-03-07 16:20:00,236 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [7d827a46] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 16:20:00,237 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-86) [3661b1b5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:20:00,237 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-86) [3661b1b5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 4:20 PM >2013-03-07 16:20:00,240 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-86) [3661b1b5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 4:20 PM >2013-03-07 16:20:02,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 70cad61 >2013-03-07 16:20:02,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) FINISH, GetHardwareInfoVDSCommand, log id: 70cad61 >2013-03-07 16:20:02,739 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-91) [41dd3945] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 16:20:02,743 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-91) [4a973cb7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 16:20:02,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4cc6a4b9 >2013-03-07 16:20:23,417 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 16:20:23,424 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:20:23,425 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:21:07,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 16:21:07,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:21:07,784 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 16:21:07,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 4cc6a4b9 >2013-03-07 16:21:07,789 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [4e35454c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:21:07,825 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [4e35454c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d50bc67 >2013-03-07 16:21:07,825 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [4e35454c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 16:21:07,825 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [4e35454c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d50bc67 >2013-03-07 16:21:07,828 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [4e35454c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 16:21:07,828 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [4a973cb7] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 16:21:07,838 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-91) [a6d398] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:21:07,840 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [a6d398] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: fbc2526 >2013-03-07 16:21:07,841 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [a6d398] FINISH, SetVdsStatusVDSCommand, log id: fbc2526 >2013-03-07 16:21:07,847 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-91) [a6d398] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 16:21:07,870 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-91) [16daca1a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:21:07,872 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-91) [906cd46] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:21:07,874 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-91) [906cd46] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 16:21:23,418 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-07 16:21:23,425 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:21:23,425 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:22:23,420 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 16:22:23,427 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:22:23,427 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:23:23,422 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 16:23:23,428 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:23:23,428 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:24:23,423 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-07 16:24:23,429 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:24:23,429 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) Autorecovering 1 hosts >2013-03-07 16:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [6e4ed78a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [6e4ed78a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [6e4ed78a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5cad8bbc >2013-03-07 16:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [6e4ed78a] FINISH, SetVdsStatusVDSCommand, log id: 5cad8bbc >2013-03-07 16:25:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [6e4ed78a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 47ca9240 >2013-03-07 16:25:00,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [6e4ed78a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3b015d91 >2013-03-07 16:25:00,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [6e4ed78a] FINISH, GetHardwareInfoVDSCommand, log id: 3b015d91 >2013-03-07 16:25:00,462 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [3d2c8454] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:25:00,464 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [1b722047] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:25:00,466 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [1b722047] FINISH, ActivateVdsVDSCommand, log id: 47ca9240 >2013-03-07 16:25:00,486 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [1b722047] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:25:00,487 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [1b722047] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 16:25:00,488 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [1b722047] Autorecovering 1 storage domains >2013-03-07 16:25:00,488 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [1b722047] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 16:25:00,489 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [4d80bb70] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:25:00,490 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [4d80bb70] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 4:25 PM >2013-03-07 16:25:00,492 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [4d80bb70] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 4:25 PM >2013-03-07 16:25:02,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 155bc585 >2013-03-07 16:25:02,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) FINISH, GetHardwareInfoVDSCommand, log id: 155bc585 >2013-03-07 16:25:02,122 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-39) [1c8d338d] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 16:25:02,126 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-39) [67cb4a15] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 16:25:02,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7761f778 >2013-03-07 16:25:23,424 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-07 16:25:23,435 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:25:23,435 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:26:06,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 16:26:06,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:26:06,121 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 16:26:06,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 7761f778 >2013-03-07 16:26:06,125 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [7696e356] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:26:06,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [7696e356] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a72cadb >2013-03-07 16:26:06,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [7696e356] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 16:26:06,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [7696e356] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a72cadb >2013-03-07 16:26:06,148 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [7696e356] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 16:26:06,148 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [67cb4a15] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 16:26:06,164 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-39) [57532647] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:26:06,165 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [57532647] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 401e3da0 >2013-03-07 16:26:06,167 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [57532647] FINISH, SetVdsStatusVDSCommand, log id: 401e3da0 >2013-03-07 16:26:06,174 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-39) [57532647] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 16:26:06,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-39) [58af85da] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:26:06,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-39) [3afc04ef] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:26:06,203 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-39) [3afc04ef] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 16:26:23,426 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-07 16:26:23,432 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:26:23,433 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:27:23,427 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-07 16:27:23,434 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:27:23,435 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:28:23,429 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-07 16:28:23,436 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:28:23,436 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:29:23,430 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-07 16:29:23,437 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:29:23,437 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) Autorecovering 1 hosts >2013-03-07 16:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [69b9839f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [69b9839f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [69b9839f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5bc32b5c >2013-03-07 16:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [69b9839f] FINISH, SetVdsStatusVDSCommand, log id: 5bc32b5c >2013-03-07 16:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [69b9839f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5eb9116a >2013-03-07 16:30:00,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [69b9839f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6f548f2e >2013-03-07 16:30:00,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [69b9839f] FINISH, GetHardwareInfoVDSCommand, log id: 6f548f2e >2013-03-07 16:30:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-84) [2bc868e3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:30:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-84) [3115f4bc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:30:00,094 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [3115f4bc] FINISH, ActivateVdsVDSCommand, log id: 5eb9116a >2013-03-07 16:30:00,121 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [3115f4bc] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:30:00,122 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-84) [3115f4bc] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 16:30:00,123 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [3115f4bc] Autorecovering 1 storage domains >2013-03-07 16:30:00,123 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [3115f4bc] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 16:30:00,124 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [467d1c57] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:30:00,125 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [467d1c57] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 4:30 PM >2013-03-07 16:30:00,127 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [467d1c57] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 4:30 PM >2013-03-07 16:30:00,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 54f7dc95 >2013-03-07 16:30:00,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) FINISH, GetHardwareInfoVDSCommand, log id: 54f7dc95 >2013-03-07 16:30:00,456 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-87) [71508822] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 16:30:00,459 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-87) [e6dc1f6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 16:30:00,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1b20f4c7 >2013-03-07 16:30:23,431 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-07 16:30:23,439 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-07 16:30:23,440 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:30:33,696 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-10) Start refreshing all users data >2013-03-07 16:30:34,547 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf. >2013-03-07 16:30:34,548 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-21) Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 16:30:34,548 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-21) Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 16:30:34,549 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-21) Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 16:30:34,549 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-21) Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 16:30:34,549 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-21) Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 16:30:34,550 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-21) Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 16:30:35,484 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-8) [2598f594] Running command: LogoutUserCommand internal: false. >2013-03-07 16:31:05,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 16:31:05,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:31:05,477 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 16:31:05,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 1b20f4c7 >2013-03-07 16:31:05,481 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [1b3a7834] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:31:05,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1b3a7834] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 302a5287 >2013-03-07 16:31:05,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [1b3a7834] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 16:31:05,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1b3a7834] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 302a5287 >2013-03-07 16:31:05,505 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [1b3a7834] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 16:31:05,510 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [e6dc1f6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 16:31:05,523 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-87) [78164438] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:31:05,525 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [78164438] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2a386ede >2013-03-07 16:31:05,526 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [78164438] FINISH, SetVdsStatusVDSCommand, log id: 2a386ede >2013-03-07 16:31:05,537 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [78164438] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 16:31:05,560 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [4a0579ce] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:31:05,562 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [4fdb94a0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:31:05,564 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-87) [4fdb94a0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 16:31:27,592 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginAdminUserCommand internal: false. >2013-03-07 16:31:27,616 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-07 16:31:28,102 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (WebAdminUpperRightButtonURL) with null version, using default general for version >2013-03-07 16:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [9a140e] Autorecovering 1 hosts >2013-03-07 16:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [9a140e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [4fbea70b] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [4fbea70b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [4fbea70b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2f8103bb >2013-03-07 16:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [4fbea70b] FINISH, SetVdsStatusVDSCommand, log id: 2f8103bb >2013-03-07 16:35:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-70) [4fbea70b] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6b7de6f4 >2013-03-07 16:35:00,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [4fbea70b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4c19dd8b >2013-03-07 16:35:00,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [4fbea70b] FINISH, GetHardwareInfoVDSCommand, log id: 4c19dd8b >2013-03-07 16:35:00,416 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-70) [c109376] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:35:00,419 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-70) [662944ab] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:35:00,420 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-70) [662944ab] FINISH, ActivateVdsVDSCommand, log id: 6b7de6f4 >2013-03-07 16:35:00,440 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [662944ab] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:35:00,441 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-70) [662944ab] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 16:35:00,442 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [662944ab] Autorecovering 1 storage domains >2013-03-07 16:35:00,442 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [662944ab] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 16:35:00,443 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [24e9d71d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:35:00,443 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [24e9d71d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 4:35 PM >2013-03-07 16:35:00,446 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [24e9d71d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 4:35 PM >2013-03-07 16:35:02,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [4d80bb70] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 29d39ef2 >2013-03-07 16:35:02,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [4d80bb70] FINISH, GetHardwareInfoVDSCommand, log id: 29d39ef2 >2013-03-07 16:35:02,813 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-36) [713f5a61] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 16:35:02,816 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-36) [7b968186] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 16:35:02,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: d3b3232 >2013-03-07 16:36:07,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 16:36:07,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:36:07,808 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 16:36:07,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: d3b3232 >2013-03-07 16:36:07,812 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [7220bfdd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:36:07,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [7220bfdd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 46226688 >2013-03-07 16:36:07,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [7220bfdd] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 16:36:07,828 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [7220bfdd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 46226688 >2013-03-07 16:36:07,830 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [7220bfdd] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 16:36:07,830 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [7b968186] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 16:36:07,840 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-36) [2fa01ac5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:36:07,842 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [2fa01ac5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2415a517 >2013-03-07 16:36:07,843 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [2fa01ac5] FINISH, SetVdsStatusVDSCommand, log id: 2415a517 >2013-03-07 16:36:07,849 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [2fa01ac5] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 16:36:07,868 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [47bff203] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:36:07,871 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [5ea344e8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:36:07,872 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-36) [5ea344e8] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 16:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) Autorecovering 1 hosts >2013-03-07 16:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-76) [71b1fa5] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-76) [71b1fa5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [71b1fa5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 47c4dea5 >2013-03-07 16:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [71b1fa5] FINISH, SetVdsStatusVDSCommand, log id: 47c4dea5 >2013-03-07 16:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-76) [71b1fa5] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 52944f98 >2013-03-07 16:40:00,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [71b1fa5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 67ae67ca >2013-03-07 16:40:00,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [71b1fa5] FINISH, GetHardwareInfoVDSCommand, log id: 67ae67ca >2013-03-07 16:40:00,096 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-76) [474ab974] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:40:00,099 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-76) [5738c4b5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:40:00,103 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-76) [5738c4b5] FINISH, ActivateVdsVDSCommand, log id: 52944f98 >2013-03-07 16:40:00,123 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-76) [5738c4b5] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:40:00,124 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-76) [5738c4b5] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 16:40:00,125 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [5738c4b5] Autorecovering 1 storage domains >2013-03-07 16:40:00,125 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [5738c4b5] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 16:40:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-76) [58eda9be] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:40:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-76) [58eda9be] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 4:40 PM >2013-03-07 16:40:00,129 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-76) [58eda9be] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 4:40 PM >2013-03-07 16:40:02,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1380b5ea >2013-03-07 16:40:02,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) FINISH, GetHardwareInfoVDSCommand, log id: 1380b5ea >2013-03-07 16:40:02,114 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-81) [1d89bc30] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 16:40:02,118 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-81) [7d6f17e8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 16:40:02,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3377488c >2013-03-07 16:41:07,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 16:41:07,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:41:07,121 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 16:41:07,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 3377488c >2013-03-07 16:41:07,125 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [989aa3e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:41:07,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [989aa3e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 26e42e3c >2013-03-07 16:41:07,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [989aa3e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 16:41:07,146 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [989aa3e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 26e42e3c >2013-03-07 16:41:07,148 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [989aa3e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 16:41:07,149 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [7d6f17e8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 16:41:07,158 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-81) [1ddd8115] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:41:07,160 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [1ddd8115] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 260c7ae0 >2013-03-07 16:41:07,161 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [1ddd8115] FINISH, SetVdsStatusVDSCommand, log id: 260c7ae0 >2013-03-07 16:41:07,168 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-81) [1ddd8115] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 16:41:07,188 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-81) [7354451a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:41:07,191 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-81) [7820c326] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:41:07,192 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-81) [7820c326] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 16:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-31) Autorecovering 1 hosts >2013-03-07 16:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-31) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-31) [14c4db24] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-31) [14c4db24] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [14c4db24] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: bdd13fb >2013-03-07 16:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [14c4db24] FINISH, SetVdsStatusVDSCommand, log id: bdd13fb >2013-03-07 16:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-31) [14c4db24] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: e33544 >2013-03-07 16:45:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [14c4db24] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1b44ffd5 >2013-03-07 16:45:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [14c4db24] FINISH, GetHardwareInfoVDSCommand, log id: 1b44ffd5 >2013-03-07 16:45:00,259 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-31) [2c7a5cb5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:45:00,261 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-31) [535af4e7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:45:00,263 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-31) [535af4e7] FINISH, ActivateVdsVDSCommand, log id: e33544 >2013-03-07 16:45:00,284 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-31) [535af4e7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:45:00,285 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-31) [535af4e7] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 16:45:00,286 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-31) [535af4e7] Autorecovering 1 storage domains >2013-03-07 16:45:00,286 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-31) [535af4e7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 16:45:00,287 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-31) [3f1e2845] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:45:00,287 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-31) [3f1e2845] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 4:45 PM >2013-03-07 16:45:00,290 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-31) [3f1e2845] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 4:45 PM >2013-03-07 16:45:01,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6a9f6aeb >2013-03-07 16:45:01,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) FINISH, GetHardwareInfoVDSCommand, log id: 6a9f6aeb >2013-03-07 16:45:01,469 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-26) [53cb50da] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 16:45:01,473 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-26) [f36a3a3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 16:45:01,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 216122fb >2013-03-07 16:46:06,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 16:46:06,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:46:06,776 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 16:46:06,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 216122fb >2013-03-07 16:46:06,780 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [6f092765] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:46:06,795 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [6f092765] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6e107567 >2013-03-07 16:46:06,796 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [6f092765] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 16:46:06,796 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [6f092765] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6e107567 >2013-03-07 16:46:06,798 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [6f092765] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 16:46:06,798 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [f36a3a3] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 16:46:06,808 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-26) [4291c3e9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:46:06,810 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [4291c3e9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6eb8b72 >2013-03-07 16:46:06,811 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [4291c3e9] FINISH, SetVdsStatusVDSCommand, log id: 6eb8b72 >2013-03-07 16:46:06,817 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [4291c3e9] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 16:46:06,841 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [66a7bb76] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:46:06,844 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [37a9b8d1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:46:06,845 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-26) [37a9b8d1] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 16:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [769d0032] Autorecovering 1 hosts >2013-03-07 16:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [769d0032] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [50576723] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [50576723] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [50576723] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 109705b1 >2013-03-07 16:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [50576723] FINISH, SetVdsStatusVDSCommand, log id: 109705b1 >2013-03-07 16:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-2) [50576723] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 76a2411a >2013-03-07 16:50:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [50576723] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 48bf26fa >2013-03-07 16:50:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [50576723] FINISH, GetHardwareInfoVDSCommand, log id: 48bf26fa >2013-03-07 16:50:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [75273a31] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:50:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [15ad0678] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:50:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-2) [15ad0678] FINISH, ActivateVdsVDSCommand, log id: 76a2411a >2013-03-07 16:50:00,224 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [15ad0678] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:50:00,225 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [15ad0678] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 16:50:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [15ad0678] Autorecovering 1 storage domains >2013-03-07 16:50:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [15ad0678] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 16:50:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [72589424] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:50:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [72589424] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 4:50 PM >2013-03-07 16:50:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [72589424] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 4:50 PM >2013-03-07 16:50:01,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5b6139d3 >2013-03-07 16:50:01,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) FINISH, GetHardwareInfoVDSCommand, log id: 5b6139d3 >2013-03-07 16:50:01,099 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-83) [46c901ac] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 16:50:01,103 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-83) [12017ab1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 16:50:01,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6d4a7816 >2013-03-07 16:51:06,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 16:51:06,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:51:06,181 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 16:51:06,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 6d4a7816 >2013-03-07 16:51:06,185 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [56ceec7a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:51:06,200 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [56ceec7a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7ecc3f0c >2013-03-07 16:51:06,200 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [56ceec7a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 16:51:06,200 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [56ceec7a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7ecc3f0c >2013-03-07 16:51:06,206 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [56ceec7a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 16:51:06,206 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [12017ab1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 16:51:06,216 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-83) [6ff74333] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:51:06,218 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [6ff74333] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 450db40e >2013-03-07 16:51:06,219 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [6ff74333] FINISH, SetVdsStatusVDSCommand, log id: 450db40e >2013-03-07 16:51:06,225 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-83) [6ff74333] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 16:51:06,245 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-83) [4b31f0bf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:51:06,248 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-83) [35a8072] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:51:06,249 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-83) [35a8072] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 16:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [3fa3ab4e] Autorecovering 1 hosts >2013-03-07 16:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [3fa3ab4e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [52c50e8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [52c50e8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [52c50e8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 14548817 >2013-03-07 16:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [52c50e8] FINISH, SetVdsStatusVDSCommand, log id: 14548817 >2013-03-07 16:55:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-18) [52c50e8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 56de86c6 >2013-03-07 16:55:00,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [52c50e8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4e83054f >2013-03-07 16:55:00,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [52c50e8] FINISH, GetHardwareInfoVDSCommand, log id: 4e83054f >2013-03-07 16:55:00,212 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [ca579f7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:55:00,215 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [6596c8d3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:55:00,216 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-18) [6596c8d3] FINISH, ActivateVdsVDSCommand, log id: 56de86c6 >2013-03-07 16:55:00,242 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [6596c8d3] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 16:55:00,243 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [6596c8d3] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 16:55:00,245 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [6596c8d3] Autorecovering 1 storage domains >2013-03-07 16:55:00,245 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [6596c8d3] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 16:55:00,246 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [57025725] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:55:00,246 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [57025725] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 4:55 PM >2013-03-07 16:55:00,249 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [57025725] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 4:55 PM >2013-03-07 16:55:00,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [42fdb2dc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 713c541e >2013-03-07 16:55:00,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [42fdb2dc] FINISH, GetHardwareInfoVDSCommand, log id: 713c541e >2013-03-07 16:55:00,486 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-59) [30a4567b] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 16:55:00,490 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-59) [56d4a7bc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 16:55:00,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 68ab7b53 >2013-03-07 16:56:04,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 16:56:04,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 16:56:04,496 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 16:56:04,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 68ab7b53 >2013-03-07 16:56:04,500 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [66cc1b18] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 16:56:04,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [66cc1b18] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 385988d1 >2013-03-07 16:56:04,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [66cc1b18] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 16:56:04,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [66cc1b18] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 385988d1 >2013-03-07 16:56:04,519 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [66cc1b18] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 16:56:04,520 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [56d4a7bc] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 16:56:04,531 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-59) [3d700e4f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:56:04,533 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [3d700e4f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 32d6289a >2013-03-07 16:56:04,534 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [3d700e4f] FINISH, SetVdsStatusVDSCommand, log id: 32d6289a >2013-03-07 16:56:04,540 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [3d700e4f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 16:56:04,560 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [7ef57e32] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:56:04,562 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [5b7cc9ec] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 16:56:04,564 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-59) [5b7cc9ec] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 17:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) Autorecovering 1 hosts >2013-03-07 17:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-67) [91a6e54] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-67) [91a6e54] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [91a6e54] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2d7780b1 >2013-03-07 17:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [91a6e54] FINISH, SetVdsStatusVDSCommand, log id: 2d7780b1 >2013-03-07 17:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-67) [91a6e54] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 419d2a7a >2013-03-07 17:00:00,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [91a6e54] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 633225e9 >2013-03-07 17:00:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [91a6e54] FINISH, GetHardwareInfoVDSCommand, log id: 633225e9 >2013-03-07 17:00:00,185 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-67) [4ca36c3e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:00:00,188 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-67) [6dd2c0ae] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:00:00,189 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-67) [6dd2c0ae] FINISH, ActivateVdsVDSCommand, log id: 419d2a7a >2013-03-07 17:00:00,209 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-67) [6dd2c0ae] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:00:00,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-67) [6dd2c0ae] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 17:00:00,211 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [6dd2c0ae] Autorecovering 1 storage domains >2013-03-07 17:00:00,211 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [6dd2c0ae] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 17:00:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-67) [136fea9d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:00:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-67) [136fea9d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 5:00 PM >2013-03-07 17:00:00,215 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-67) [136fea9d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 5:00 PM >2013-03-07 17:00:01,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-71) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 653d425c >2013-03-07 17:00:01,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-71) FINISH, GetHardwareInfoVDSCommand, log id: 653d425c >2013-03-07 17:00:01,812 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-71) [132cd40a] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 17:00:01,816 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-71) [71b617c6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 17:00:01,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: d9f6f0c >2013-03-07 17:01:06,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 17:01:06,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:01:06,822 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 17:01:06,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: d9f6f0c >2013-03-07 17:01:06,828 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [14933ca4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:01:06,846 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [14933ca4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 37802665 >2013-03-07 17:01:06,846 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [14933ca4] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 17:01:06,846 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [14933ca4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 37802665 >2013-03-07 17:01:06,848 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [14933ca4] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 17:01:06,848 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [71b617c6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 17:01:06,860 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-71) [38116851] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:01:06,862 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-71) [38116851] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 23088cb >2013-03-07 17:01:06,863 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-71) [38116851] FINISH, SetVdsStatusVDSCommand, log id: 23088cb >2013-03-07 17:01:06,869 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-71) [38116851] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 17:01:06,889 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-71) [79d70615] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:01:06,892 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-71) [1b4c1700] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:01:06,893 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-71) [1b4c1700] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 17:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) Autorecovering 1 hosts >2013-03-07 17:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-12) [1094e2ce] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-12) [1094e2ce] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [1094e2ce] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4db23a64 >2013-03-07 17:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [1094e2ce] FINISH, SetVdsStatusVDSCommand, log id: 4db23a64 >2013-03-07 17:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-12) [1094e2ce] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6e35894b >2013-03-07 17:05:00,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [1094e2ce] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 446006a >2013-03-07 17:05:00,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [1094e2ce] FINISH, GetHardwareInfoVDSCommand, log id: 446006a >2013-03-07 17:05:00,406 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-12) [16a9a383] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:05:00,409 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-12) [159f65] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:05:00,410 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-12) [159f65] FINISH, ActivateVdsVDSCommand, log id: 6e35894b >2013-03-07 17:05:00,430 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-12) [159f65] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:05:00,431 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-12) [159f65] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 17:05:00,432 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [159f65] Autorecovering 1 storage domains >2013-03-07 17:05:00,432 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [159f65] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 17:05:00,433 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-12) [5bdf326d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:05:00,433 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-12) [5bdf326d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 5:05 PM >2013-03-07 17:05:00,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-12) [5bdf326d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 5:05 PM >2013-03-07 17:05:01,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3a1b0367 >2013-03-07 17:05:01,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) FINISH, GetHardwareInfoVDSCommand, log id: 3a1b0367 >2013-03-07 17:05:01,138 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-17) [5af57cf3] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 17:05:01,142 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-17) [6ecef7f8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 17:05:01,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 62f5887 >2013-03-07 17:06:05,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 17:06:05,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:06:05,444 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 17:06:05,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 62f5887 >2013-03-07 17:06:05,448 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [20a344d1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:06:05,467 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [20a344d1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 693a8e23 >2013-03-07 17:06:05,467 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [20a344d1] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 17:06:05,468 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [20a344d1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 693a8e23 >2013-03-07 17:06:05,473 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [6ecef7f8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 17:06:05,489 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-17) [3031409c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:06:05,491 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [3031409c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7303455 >2013-03-07 17:06:05,492 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [3031409c] FINISH, SetVdsStatusVDSCommand, log id: 7303455 >2013-03-07 17:06:05,497 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [3031409c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 17:06:05,516 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [37ab4b88] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:06:05,518 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [6f2114a2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:06:05,520 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-17) [6f2114a2] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 17:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-66) Autorecovering 1 hosts >2013-03-07 17:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-66) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-66) [73c5623c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-66) [73c5623c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [73c5623c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 269ca22f >2013-03-07 17:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [73c5623c] FINISH, SetVdsStatusVDSCommand, log id: 269ca22f >2013-03-07 17:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-66) [73c5623c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1d5ff876 >2013-03-07 17:10:00,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [73c5623c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6b37d48a >2013-03-07 17:10:00,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [73c5623c] FINISH, GetHardwareInfoVDSCommand, log id: 6b37d48a >2013-03-07 17:10:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-66) [2ef9f112] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:10:00,095 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-66) [1468dec3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:10:00,097 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-66) [1468dec3] FINISH, ActivateVdsVDSCommand, log id: 1d5ff876 >2013-03-07 17:10:00,116 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-66) [1468dec3] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:10:00,117 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-66) [1468dec3] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 17:10:00,119 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-66) [1468dec3] Autorecovering 1 storage domains >2013-03-07 17:10:00,119 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-66) [1468dec3] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 17:10:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-66) [7b0772eb] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:10:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-66) [7b0772eb] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 5:10 PM >2013-03-07 17:10:00,122 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-66) [7b0772eb] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 5:10 PM >2013-03-07 17:10:02,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 37190468 >2013-03-07 17:10:02,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) FINISH, GetHardwareInfoVDSCommand, log id: 37190468 >2013-03-07 17:10:02,774 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-65) [4160bece] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 17:10:02,778 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-65) [1e776711] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 17:10:02,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4d1a0738 >2013-03-07 17:11:07,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 17:11:07,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:11:07,757 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 17:11:07,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 4d1a0738 >2013-03-07 17:11:07,761 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [64b3f711] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:11:07,781 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [64b3f711] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1587580e >2013-03-07 17:11:07,781 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [64b3f711] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 17:11:07,781 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [64b3f711] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1587580e >2013-03-07 17:11:07,783 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [64b3f711] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 17:11:07,783 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [1e776711] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 17:11:07,793 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-65) [4511ca08] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:11:07,795 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [4511ca08] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 23cafb8b >2013-03-07 17:11:07,796 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [4511ca08] FINISH, SetVdsStatusVDSCommand, log id: 23cafb8b >2013-03-07 17:11:07,802 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-65) [4511ca08] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 17:11:07,823 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-65) [1a3b0ccf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:11:07,825 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-65) [72c8fd03] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:11:07,827 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-65) [72c8fd03] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 17:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) Autorecovering 1 hosts >2013-03-07 17:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [66e8c48d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [66e8c48d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [66e8c48d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4cd3bd22 >2013-03-07 17:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [66e8c48d] FINISH, SetVdsStatusVDSCommand, log id: 4cd3bd22 >2013-03-07 17:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [66e8c48d] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4f47c3b8 >2013-03-07 17:15:00,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [66e8c48d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6640936 >2013-03-07 17:15:00,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [66e8c48d] FINISH, GetHardwareInfoVDSCommand, log id: 6640936 >2013-03-07 17:15:00,289 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [23af105c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:15:00,292 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [42a425dd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:15:00,293 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [42a425dd] FINISH, ActivateVdsVDSCommand, log id: 4f47c3b8 >2013-03-07 17:15:00,313 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [42a425dd] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:15:00,314 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-8) [42a425dd] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 17:15:00,315 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [42a425dd] Autorecovering 1 storage domains >2013-03-07 17:15:00,315 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [42a425dd] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 17:15:00,316 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [329a0098] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:15:00,316 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [329a0098] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 5:15 PM >2013-03-07 17:15:00,319 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [329a0098] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 5:15 PM >2013-03-07 17:15:02,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [35d5810] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 9cb0598 >2013-03-07 17:15:02,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [35d5810] FINISH, GetHardwareInfoVDSCommand, log id: 9cb0598 >2013-03-07 17:15:02,078 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-52) [4375071d] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 17:15:02,082 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-52) [239d1802] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 17:15:02,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 47acb51 >2013-03-07 17:16:06,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 17:16:06,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:16:06,054 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 17:16:06,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 47acb51 >2013-03-07 17:16:06,058 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [6d47edbe] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:16:06,074 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [6d47edbe] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 27508056 >2013-03-07 17:16:06,074 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [6d47edbe] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 17:16:06,074 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [6d47edbe] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 27508056 >2013-03-07 17:16:06,076 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [6d47edbe] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 17:16:06,077 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [239d1802] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 17:16:06,086 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-52) [e973cef] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:16:06,088 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [e973cef] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 11771b91 >2013-03-07 17:16:06,089 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [e973cef] FINISH, SetVdsStatusVDSCommand, log id: 11771b91 >2013-03-07 17:16:06,095 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-52) [e973cef] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 17:16:06,114 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [511148e8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:16:06,117 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [772e1532] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:16:06,118 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-52) [772e1532] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 17:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) Autorecovering 1 hosts >2013-03-07 17:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-51) [38cfa27b] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-51) [38cfa27b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [38cfa27b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1ebe1536 >2013-03-07 17:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [38cfa27b] FINISH, SetVdsStatusVDSCommand, log id: 1ebe1536 >2013-03-07 17:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-51) [38cfa27b] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: bcd44c9 >2013-03-07 17:20:00,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [38cfa27b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4668ae69 >2013-03-07 17:20:00,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [38cfa27b] FINISH, GetHardwareInfoVDSCommand, log id: 4668ae69 >2013-03-07 17:20:00,264 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-51) [7ff72857] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:20:00,267 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-51) [6daa6958] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:20:00,269 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-51) [6daa6958] FINISH, ActivateVdsVDSCommand, log id: bcd44c9 >2013-03-07 17:20:00,289 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-51) [6daa6958] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:20:00,290 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-51) [6daa6958] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 17:20:00,291 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [6daa6958] Autorecovering 1 storage domains >2013-03-07 17:20:00,291 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [6daa6958] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 17:20:00,292 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-51) [5be7886b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:20:00,292 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-51) [5be7886b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 5:20 PM >2013-03-07 17:20:00,296 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-51) [5be7886b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 5:20 PM >2013-03-07 17:20:00,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [906cd46] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 761309cd >2013-03-07 17:20:00,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [906cd46] FINISH, GetHardwareInfoVDSCommand, log id: 761309cd >2013-03-07 17:20:00,368 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-91) [71155372] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 17:20:00,372 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-91) [62febdf1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 17:20:00,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4ccee925 >2013-03-07 17:21:06,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 17:21:06,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:21:06,405 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 17:21:06,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 4ccee925 >2013-03-07 17:21:06,409 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [7e7002c0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:21:06,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [7e7002c0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5ad18691 >2013-03-07 17:21:06,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [7e7002c0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 17:21:06,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [7e7002c0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5ad18691 >2013-03-07 17:21:06,427 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [7e7002c0] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 17:21:06,427 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [62febdf1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 17:21:06,437 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-91) [193dc071] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:21:06,439 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [193dc071] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 60ad3f4a >2013-03-07 17:21:06,440 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [193dc071] FINISH, SetVdsStatusVDSCommand, log id: 60ad3f4a >2013-03-07 17:21:06,447 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-91) [193dc071] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 17:21:06,471 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-91) [1f4fb4b6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:21:06,474 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-91) [3742243e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:21:06,475 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-91) [3742243e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 17:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) Autorecovering 1 hosts >2013-03-07 17:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-97) [6e7673d4] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-97) [6e7673d4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [6e7673d4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6bbc8091 >2013-03-07 17:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [6e7673d4] FINISH, SetVdsStatusVDSCommand, log id: 6bbc8091 >2013-03-07 17:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-97) [6e7673d4] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 11534f0 >2013-03-07 17:25:00,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [6e7673d4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 795b3471 >2013-03-07 17:25:00,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [6e7673d4] FINISH, GetHardwareInfoVDSCommand, log id: 795b3471 >2013-03-07 17:25:00,188 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-97) [7ae806d4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:25:00,191 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-97) [24aff10e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:25:00,192 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-97) [24aff10e] FINISH, ActivateVdsVDSCommand, log id: 11534f0 >2013-03-07 17:25:00,212 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-97) [24aff10e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:25:00,213 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-97) [24aff10e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 17:25:00,215 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) [24aff10e] Autorecovering 1 storage domains >2013-03-07 17:25:00,215 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) [24aff10e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 17:25:00,216 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-97) [1e658023] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:25:00,216 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-97) [1e658023] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 5:25 PM >2013-03-07 17:25:00,219 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-97) [1e658023] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 5:25 PM >2013-03-07 17:25:00,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 50bce428 >2013-03-07 17:25:00,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) FINISH, GetHardwareInfoVDSCommand, log id: 50bce428 >2013-03-07 17:25:00,726 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-4) [550a7bb6] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 17:25:00,730 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-4) [7048497e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 17:25:00,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7a2c5cd9 >2013-03-07 17:26:06,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 17:26:06,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:26:06,732 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 17:26:06,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 7a2c5cd9 >2013-03-07 17:26:06,735 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [1ff08231] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:26:06,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1ff08231] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 676f1c53 >2013-03-07 17:26:06,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [1ff08231] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 17:26:06,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1ff08231] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 676f1c53 >2013-03-07 17:26:06,754 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [1ff08231] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 17:26:06,754 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [7048497e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 17:26:06,764 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-4) [2d27ad77] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:26:06,766 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [2d27ad77] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: e318e38 >2013-03-07 17:26:06,767 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [2d27ad77] FINISH, SetVdsStatusVDSCommand, log id: e318e38 >2013-03-07 17:26:06,772 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-4) [2d27ad77] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 17:26:06,792 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-4) [7ea290b9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:26:06,794 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-4) [1e67011c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:26:06,796 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-4) [1e67011c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 17:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) Autorecovering 1 hosts >2013-03-07 17:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [2115d134] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [2115d134] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [2115d134] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 9f443ae >2013-03-07 17:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [2115d134] FINISH, SetVdsStatusVDSCommand, log id: 9f443ae >2013-03-07 17:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [2115d134] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7ee4f82f >2013-03-07 17:30:00,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [2115d134] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 762c3fe8 >2013-03-07 17:30:00,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [2115d134] FINISH, GetHardwareInfoVDSCommand, log id: 762c3fe8 >2013-03-07 17:30:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [17c26ba5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:30:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [4ac329be] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:30:00,206 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [4ac329be] FINISH, ActivateVdsVDSCommand, log id: 7ee4f82f >2013-03-07 17:30:00,226 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [4ac329be] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:30:00,227 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [4ac329be] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 17:30:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [4ac329be] Autorecovering 1 storage domains >2013-03-07 17:30:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [4ac329be] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 17:30:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [6c811f72] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:30:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [6c811f72] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 5:30 PM >2013-03-07 17:30:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [6c811f72] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 5:30 PM >2013-03-07 17:30:00,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3cec29b9 >2013-03-07 17:30:00,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) FINISH, GetHardwareInfoVDSCommand, log id: 3cec29b9 >2013-03-07 17:30:01,034 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-47) [6d2446b7] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 17:30:01,038 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-47) [efe8a00] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 17:30:01,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1954fa9d >2013-03-07 17:30:33,698 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-76) [58eda9be] Start refreshing all users data >2013-03-07 17:30:34,551 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [6f2114a2] Attempting to update VMs/Templates Ovf. >2013-03-07 17:30:34,552 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [6f2114a2] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 17:30:34,559 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [6f2114a2] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 17:30:34,559 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [6f2114a2] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 17:30:34,561 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [6f2114a2] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 17:30:34,562 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [6f2114a2] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 17:30:34,563 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [6f2114a2] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 17:31:05,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 17:31:05,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:31:05,017 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 17:31:05,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 1954fa9d >2013-03-07 17:31:05,021 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [169ebee4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:31:05,044 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [169ebee4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3581255 >2013-03-07 17:31:05,044 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [169ebee4] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 17:31:05,044 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [169ebee4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3581255 >2013-03-07 17:31:05,047 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [169ebee4] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 17:31:05,047 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [efe8a00] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 17:31:05,058 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-47) [4833ceaa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:31:05,059 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [4833ceaa] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 60cdc1bd >2013-03-07 17:31:05,061 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [4833ceaa] FINISH, SetVdsStatusVDSCommand, log id: 60cdc1bd >2013-03-07 17:31:05,067 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [4833ceaa] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 17:31:05,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [27fd296f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:31:05,091 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [4b2ef732] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:31:05,093 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-47) [4b2ef732] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 17:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) Autorecovering 1 hosts >2013-03-07 17:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-100) [6e0e6d5e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-100) [6e0e6d5e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [6e0e6d5e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 375dd9c8 >2013-03-07 17:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [6e0e6d5e] FINISH, SetVdsStatusVDSCommand, log id: 375dd9c8 >2013-03-07 17:35:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-100) [6e0e6d5e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5f4d42ae >2013-03-07 17:35:00,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [6e0e6d5e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 791fdd8f >2013-03-07 17:35:00,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [6e0e6d5e] FINISH, GetHardwareInfoVDSCommand, log id: 791fdd8f >2013-03-07 17:35:00,219 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-100) [3d5b32c7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:35:00,222 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-100) [7b0ef153] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:35:00,224 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-100) [7b0ef153] FINISH, ActivateVdsVDSCommand, log id: 5f4d42ae >2013-03-07 17:35:00,246 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-100) [7b0ef153] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:35:00,247 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-100) [7b0ef153] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 17:35:00,248 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [7b0ef153] Autorecovering 1 storage domains >2013-03-07 17:35:00,248 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [7b0ef153] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 17:35:00,249 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-100) [24e13c2f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:35:00,249 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-100) [24e13c2f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 5:35 PM >2013-03-07 17:35:00,252 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-100) [24e13c2f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 5:35 PM >2013-03-07 17:35:02,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [3afc04ef] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 491932da >2013-03-07 17:35:02,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [3afc04ef] FINISH, GetHardwareInfoVDSCommand, log id: 491932da >2013-03-07 17:35:02,341 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-39) [6f4bc3bc] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 17:35:02,345 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-39) [48cba103] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 17:35:02,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2db777ae >2013-03-07 17:36:08,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 17:36:08,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:36:08,331 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 17:36:08,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 2db777ae >2013-03-07 17:36:08,335 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [321f4a6c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:36:08,352 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [321f4a6c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 57ad72a9 >2013-03-07 17:36:08,352 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [321f4a6c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 17:36:08,352 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [321f4a6c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 57ad72a9 >2013-03-07 17:36:08,355 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [321f4a6c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 17:36:08,355 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [48cba103] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 17:36:08,365 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-39) [4f83778a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:36:08,367 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [4f83778a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1414f9a9 >2013-03-07 17:36:08,368 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [4f83778a] FINISH, SetVdsStatusVDSCommand, log id: 1414f9a9 >2013-03-07 17:36:08,374 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-39) [4f83778a] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 17:36:08,396 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-39) [3f473d24] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:36:08,398 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-39) [5e3d6e68] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:36:08,400 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-39) [5e3d6e68] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 17:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) Autorecovering 1 hosts >2013-03-07 17:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [1204f28a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [1204f28a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [1204f28a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 71017d9b >2013-03-07 17:40:00,017 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [1204f28a] FINISH, SetVdsStatusVDSCommand, log id: 71017d9b >2013-03-07 17:40:00,020 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-50) [1204f28a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: b6117ed >2013-03-07 17:40:00,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [1204f28a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 369706b6 >2013-03-07 17:40:00,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [1204f28a] FINISH, GetHardwareInfoVDSCommand, log id: 369706b6 >2013-03-07 17:40:00,224 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-50) [32638e43] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:40:00,227 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-50) [6c84bf40] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:40:00,228 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-50) [6c84bf40] FINISH, ActivateVdsVDSCommand, log id: b6117ed >2013-03-07 17:40:00,248 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [6c84bf40] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:40:00,249 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-50) [6c84bf40] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 17:40:00,250 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [6c84bf40] Autorecovering 1 storage domains >2013-03-07 17:40:00,250 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [6c84bf40] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 17:40:00,251 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [5f57ee5e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:40:00,251 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [5f57ee5e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 5:40 PM >2013-03-07 17:40:00,254 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [5f57ee5e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 5:40 PM >2013-03-07 17:40:02,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [7820c326] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 483026ed >2013-03-07 17:40:02,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [7820c326] FINISH, GetHardwareInfoVDSCommand, log id: 483026ed >2013-03-07 17:40:02,647 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-81) [532a0095] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 17:40:02,650 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-81) [db9285b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 17:40:02,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 72e9fe94 >2013-03-07 17:41:06,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 17:41:06,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:41:06,749 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 17:41:06,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 72e9fe94 >2013-03-07 17:41:06,753 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [428e3d49] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:41:06,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [428e3d49] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 753a49ca >2013-03-07 17:41:06,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [428e3d49] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 17:41:06,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [428e3d49] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 753a49ca >2013-03-07 17:41:06,772 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [428e3d49] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 17:41:06,772 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [db9285b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 17:41:06,783 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-81) [3bdf51a0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:41:06,784 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [3bdf51a0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 58ed7bf2 >2013-03-07 17:41:06,786 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [3bdf51a0] FINISH, SetVdsStatusVDSCommand, log id: 58ed7bf2 >2013-03-07 17:41:06,792 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-81) [3bdf51a0] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 17:41:06,812 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-81) [4748a67f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:41:06,815 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-81) [1760c683] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:41:06,816 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-81) [1760c683] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 17:45:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) Autorecovering 1 hosts >2013-03-07 17:45:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:45:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [a02721a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:45:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [a02721a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [a02721a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 8a6ff5a >2013-03-07 17:45:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [a02721a] FINISH, SetVdsStatusVDSCommand, log id: 8a6ff5a >2013-03-07 17:45:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-92) [a02721a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 54d470d0 >2013-03-07 17:45:00,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [a02721a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 77d0b014 >2013-03-07 17:45:00,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [a02721a] FINISH, GetHardwareInfoVDSCommand, log id: 77d0b014 >2013-03-07 17:45:00,278 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-92) [522197a5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:45:00,280 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-92) [4887c468] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:45:00,281 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-92) [4887c468] FINISH, ActivateVdsVDSCommand, log id: 54d470d0 >2013-03-07 17:45:00,301 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [4887c468] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:45:00,302 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-92) [4887c468] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 17:45:00,304 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [4887c468] Autorecovering 1 storage domains >2013-03-07 17:45:00,304 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [4887c468] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 17:45:00,305 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [7fbdf728] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:45:00,305 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [7fbdf728] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 5:45 PM >2013-03-07 17:45:00,311 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [7fbdf728] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 5:45 PM >2013-03-07 17:45:01,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [37a9b8d1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7f5853cc >2013-03-07 17:45:01,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [37a9b8d1] FINISH, GetHardwareInfoVDSCommand, log id: 7f5853cc >2013-03-07 17:45:01,084 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-26) [46c2cc1f] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 17:45:01,088 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-26) [548bb830] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 17:45:01,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2649542c >2013-03-07 17:46:06,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 17:46:06,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:46:06,076 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 17:46:06,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 2649542c >2013-03-07 17:46:06,080 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [3af4d9a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:46:06,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [3af4d9a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 312fe6c9 >2013-03-07 17:46:06,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [3af4d9a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 17:46:06,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [3af4d9a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 312fe6c9 >2013-03-07 17:46:06,100 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [3af4d9a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 17:46:06,100 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [548bb830] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 17:46:06,111 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-26) [213f77c9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:46:06,116 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [213f77c9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 487e2aa7 >2013-03-07 17:46:06,117 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [213f77c9] FINISH, SetVdsStatusVDSCommand, log id: 487e2aa7 >2013-03-07 17:46:06,125 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [213f77c9] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 17:46:06,162 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [6110cc5b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:46:06,164 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [30d9a407] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:46:06,166 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-26) [30d9a407] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 17:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [35a8072] Autorecovering 1 hosts >2013-03-07 17:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [35a8072] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [5011cd0c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [5011cd0c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [5011cd0c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 187c147c >2013-03-07 17:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [5011cd0c] FINISH, SetVdsStatusVDSCommand, log id: 187c147c >2013-03-07 17:50:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-83) [5011cd0c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5b148668 >2013-03-07 17:50:00,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [5011cd0c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2e8127d8 >2013-03-07 17:50:00,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [5011cd0c] FINISH, GetHardwareInfoVDSCommand, log id: 2e8127d8 >2013-03-07 17:50:00,422 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-83) [1df31c70] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:50:00,424 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-83) [2f16383d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:50:00,426 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-83) [2f16383d] FINISH, ActivateVdsVDSCommand, log id: 5b148668 >2013-03-07 17:50:00,446 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [2f16383d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:50:00,447 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-83) [2f16383d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 17:50:00,448 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [2f16383d] Autorecovering 1 storage domains >2013-03-07 17:50:00,449 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [2f16383d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 17:50:00,449 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [700a62ec] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:50:00,450 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [700a62ec] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 5:50 PM >2013-03-07 17:50:00,453 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [700a62ec] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 5:50 PM >2013-03-07 17:50:00,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [30286d96] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 10938b7c >2013-03-07 17:50:00,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [30286d96] FINISH, GetHardwareInfoVDSCommand, log id: 10938b7c >2013-03-07 17:50:00,524 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-72) [53fd0fd8] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 17:50:00,527 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-72) [2ddf1cf3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 17:50:00,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 664a538e >2013-03-07 17:51:06,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 17:51:06,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:51:06,561 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 17:51:06,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 664a538e >2013-03-07 17:51:06,565 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [299f176e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:51:06,582 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [299f176e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2ef1142a >2013-03-07 17:51:06,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [299f176e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 17:51:06,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [299f176e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2ef1142a >2013-03-07 17:51:06,585 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [299f176e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 17:51:06,585 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [2ddf1cf3] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 17:51:06,595 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-72) [cb11094] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:51:06,596 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [cb11094] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5da11517 >2013-03-07 17:51:06,598 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [cb11094] FINISH, SetVdsStatusVDSCommand, log id: 5da11517 >2013-03-07 17:51:06,604 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [cb11094] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 17:51:06,626 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [663d0da3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:51:06,629 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [438bac47] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:51:06,631 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-72) [438bac47] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 17:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) Autorecovering 1 hosts >2013-03-07 17:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:55:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [14e0dc3a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:55:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [14e0dc3a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:55:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [14e0dc3a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 158dd04c >2013-03-07 17:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [14e0dc3a] FINISH, SetVdsStatusVDSCommand, log id: 158dd04c >2013-03-07 17:55:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [14e0dc3a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2e5be28a >2013-03-07 17:55:00,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [14e0dc3a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 560dc347 >2013-03-07 17:55:00,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [14e0dc3a] FINISH, GetHardwareInfoVDSCommand, log id: 560dc347 >2013-03-07 17:55:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-79) [566dad88] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:55:00,094 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-79) [44635288] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:55:00,096 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [44635288] FINISH, ActivateVdsVDSCommand, log id: 2e5be28a >2013-03-07 17:55:00,115 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [44635288] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 17:55:00,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-79) [44635288] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 17:55:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [44635288] Autorecovering 1 storage domains >2013-03-07 17:55:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [44635288] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 17:55:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [7c43979f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:55:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [7c43979f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 5:55 PM >2013-03-07 17:55:00,125 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [7c43979f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 5:55 PM >2013-03-07 17:55:00,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 48a2b2d1 >2013-03-07 17:55:00,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) FINISH, GetHardwareInfoVDSCommand, log id: 48a2b2d1 >2013-03-07 17:55:00,902 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-82) [2b4682ac] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 17:55:00,906 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-82) [76be04fe] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 17:55:00,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6315c81d >2013-03-07 17:56:04,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 17:56:04,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 17:56:04,925 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 17:56:04,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 6315c81d >2013-03-07 17:56:04,929 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [7a3a21d3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 17:56:04,944 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [7a3a21d3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2552d709 >2013-03-07 17:56:04,944 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [7a3a21d3] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 17:56:04,944 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [7a3a21d3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2552d709 >2013-03-07 17:56:04,947 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [7a3a21d3] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 17:56:04,947 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [76be04fe] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 17:56:04,957 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-82) [4726c5d8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:56:04,958 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [4726c5d8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 44b79453 >2013-03-07 17:56:04,960 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [4726c5d8] FINISH, SetVdsStatusVDSCommand, log id: 44b79453 >2013-03-07 17:56:04,967 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-82) [4726c5d8] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 17:56:04,987 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-82) [77dd7e81] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:56:04,989 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-82) [2466e97f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 17:56:04,991 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-82) [2466e97f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 18:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) Autorecovering 1 hosts >2013-03-07 18:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-34) [6fce0a59] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-34) [6fce0a59] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [6fce0a59] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5f09d051 >2013-03-07 18:00:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [6fce0a59] FINISH, SetVdsStatusVDSCommand, log id: 5f09d051 >2013-03-07 18:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-34) [6fce0a59] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2daeaf44 >2013-03-07 18:00:00,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [6fce0a59] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7109a6a0 >2013-03-07 18:00:00,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [6fce0a59] FINISH, GetHardwareInfoVDSCommand, log id: 7109a6a0 >2013-03-07 18:00:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-34) [4e9aa88b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:00:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-34) [90c956a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:00:00,094 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-34) [90c956a] FINISH, ActivateVdsVDSCommand, log id: 2daeaf44 >2013-03-07 18:00:00,114 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-34) [90c956a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:00:00,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-34) [90c956a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 18:00:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) [90c956a] Autorecovering 1 storage domains >2013-03-07 18:00:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) [90c956a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 18:00:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-34) [182032a1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:00:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-34) [182032a1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 6:00 PM >2013-03-07 18:00:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-34) [182032a1] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 6:00 PM >2013-03-07 18:00:02,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [24e9d71d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 42089dc9 >2013-03-07 18:00:02,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [24e9d71d] FINISH, GetHardwareInfoVDSCommand, log id: 42089dc9 >2013-03-07 18:00:02,233 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-70) [448cbf38] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 18:00:02,236 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-70) [3d518658] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 18:00:02,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4a86735b >2013-03-07 18:01:07,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 18:01:07,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:01:07,224 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 18:01:07,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 4a86735b >2013-03-07 18:01:07,228 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [5e63c852] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:01:07,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5e63c852] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 360e81a0 >2013-03-07 18:01:07,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [5e63c852] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 18:01:07,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5e63c852] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 360e81a0 >2013-03-07 18:01:07,245 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [5e63c852] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 18:01:07,246 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [3d518658] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 18:01:07,256 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-70) [65a195cd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:01:07,257 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [65a195cd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1633327e >2013-03-07 18:01:07,259 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [65a195cd] FINISH, SetVdsStatusVDSCommand, log id: 1633327e >2013-03-07 18:01:07,264 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-70) [65a195cd] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 18:01:07,283 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-70) [360a381e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:01:07,286 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-70) [1589ff66] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:01:07,288 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-70) [1589ff66] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 18:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [58eda9be] Autorecovering 1 hosts >2013-03-07 18:05:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [58eda9be] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:05:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-76) [2d886b72] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:05:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-76) [2d886b72] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [2d886b72] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 242dd40a >2013-03-07 18:05:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [2d886b72] FINISH, SetVdsStatusVDSCommand, log id: 242dd40a >2013-03-07 18:05:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-76) [2d886b72] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5e6ff2c8 >2013-03-07 18:05:00,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [2d886b72] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 515218fa >2013-03-07 18:05:00,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [2d886b72] FINISH, GetHardwareInfoVDSCommand, log id: 515218fa >2013-03-07 18:05:00,096 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-76) [55718635] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:05:00,098 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-76) [1b0504aa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:05:00,100 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-76) [1b0504aa] FINISH, ActivateVdsVDSCommand, log id: 5e6ff2c8 >2013-03-07 18:05:00,121 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-76) [1b0504aa] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:05:00,122 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-76) [1b0504aa] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 18:05:00,124 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [1b0504aa] Autorecovering 1 storage domains >2013-03-07 18:05:00,125 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [1b0504aa] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 18:05:00,125 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-76) [16d2016b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:05:00,125 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-76) [16d2016b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 6:05 PM >2013-03-07 18:05:00,131 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-76) [16d2016b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 6:05 PM >2013-03-07 18:05:01,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [5b7cc9ec] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 26e0bf59 >2013-03-07 18:05:01,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [5b7cc9ec] FINISH, GetHardwareInfoVDSCommand, log id: 26e0bf59 >2013-03-07 18:05:01,534 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-59) [3b2df93e] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 18:05:01,538 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-59) [1a0ecd2a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 18:05:01,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 69f583 >2013-03-07 18:06:05,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 18:06:05,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:06:05,832 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 18:06:05,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 69f583 >2013-03-07 18:06:05,836 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [4f3b8f7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:06:05,854 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4f3b8f7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7a50b063 >2013-03-07 18:06:05,854 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [4f3b8f7] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 18:06:05,854 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4f3b8f7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7a50b063 >2013-03-07 18:06:05,859 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [1a0ecd2a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 18:06:05,873 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-59) [1bdd1e18] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:06:05,874 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [1bdd1e18] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 534332aa >2013-03-07 18:06:05,876 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [1bdd1e18] FINISH, SetVdsStatusVDSCommand, log id: 534332aa >2013-03-07 18:06:05,883 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [1bdd1e18] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 18:06:05,901 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [3a2df902] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:06:05,903 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [eaad4a6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:06:05,905 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-59) [eaad4a6] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 18:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [236f4673] Autorecovering 1 hosts >2013-03-07 18:10:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [236f4673] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [18985bd4] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [18985bd4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [18985bd4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 10796bfd >2013-03-07 18:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [18985bd4] FINISH, SetVdsStatusVDSCommand, log id: 10796bfd >2013-03-07 18:10:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-58) [18985bd4] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 61334645 >2013-03-07 18:10:00,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [18985bd4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 757e68fb >2013-03-07 18:10:00,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [18985bd4] FINISH, GetHardwareInfoVDSCommand, log id: 757e68fb >2013-03-07 18:10:00,215 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-58) [4925d19d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:10:00,217 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-58) [1fdef6c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:10:00,219 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-58) [1fdef6c] FINISH, ActivateVdsVDSCommand, log id: 61334645 >2013-03-07 18:10:00,239 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [1fdef6c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:10:00,240 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-58) [1fdef6c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 18:10:00,241 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [1fdef6c] Autorecovering 1 storage domains >2013-03-07 18:10:00,241 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [1fdef6c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 18:10:00,242 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [46564268] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:10:00,242 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [46564268] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 6:10 PM >2013-03-07 18:10:00,245 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [46564268] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 6:10 PM >2013-03-07 18:10:00,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 289941af >2013-03-07 18:10:00,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) FINISH, GetHardwareInfoVDSCommand, log id: 289941af >2013-03-07 18:10:00,316 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-28) [78c5a562] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 18:10:00,320 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-28) [74ca7fd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 18:10:00,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 27f1469c >2013-03-07 18:11:06,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 18:11:06,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:11:06,343 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 18:11:06,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 27f1469c >2013-03-07 18:11:06,347 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [55e5049d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:11:06,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [55e5049d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6a3220c8 >2013-03-07 18:11:06,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [55e5049d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 18:11:06,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [55e5049d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6a3220c8 >2013-03-07 18:11:06,364 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [55e5049d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 18:11:06,364 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [74ca7fd] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 18:11:06,374 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-28) [95a8ec6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:11:06,376 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [95a8ec6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4d603493 >2013-03-07 18:11:06,377 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [95a8ec6] FINISH, SetVdsStatusVDSCommand, log id: 4d603493 >2013-03-07 18:11:06,383 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-28) [95a8ec6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 18:11:06,404 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-28) [55042ef7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:11:06,406 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-28) [68139580] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:11:06,408 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-28) [68139580] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 18:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [772e1532] Autorecovering 1 hosts >2013-03-07 18:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [772e1532] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [38b56090] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [38b56090] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [38b56090] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5c8e6e9a >2013-03-07 18:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [38b56090] FINISH, SetVdsStatusVDSCommand, log id: 5c8e6e9a >2013-03-07 18:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [38b56090] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 559cb4b0 >2013-03-07 18:15:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [38b56090] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4c0d5b55 >2013-03-07 18:15:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [38b56090] FINISH, GetHardwareInfoVDSCommand, log id: 4c0d5b55 >2013-03-07 18:15:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [5e3afef5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:15:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [4a3e4d95] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:15:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [4a3e4d95] FINISH, ActivateVdsVDSCommand, log id: 559cb4b0 >2013-03-07 18:15:00,222 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [4a3e4d95] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:15:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [4a3e4d95] Autorecovering 1 storage domains >2013-03-07 18:15:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [4a3e4d95] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 18:15:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [75dd7f53] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:15:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [75dd7f53] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 6:15 PM >2013-03-07 18:15:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [75dd7f53] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 6:15 PM >2013-03-07 18:15:00,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 55fcdb48 >2013-03-07 18:15:00,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) FINISH, GetHardwareInfoVDSCommand, log id: 55fcdb48 >2013-03-07 18:15:00,649 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-80) [6b2597f] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 18:15:00,652 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-80) [3c2ef7d8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 18:15:00,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5a7ee5f9 >2013-03-07 18:16:05,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 18:16:05,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:16:05,637 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 18:16:05,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 5a7ee5f9 >2013-03-07 18:16:05,641 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [1ab66b3c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:16:05,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1ab66b3c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5ed4f09 >2013-03-07 18:16:05,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [1ab66b3c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 18:16:05,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1ab66b3c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5ed4f09 >2013-03-07 18:16:05,659 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [1ab66b3c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 18:16:05,659 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [3c2ef7d8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 18:16:05,669 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-80) [4924927d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:16:05,671 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [4924927d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2a1f6742 >2013-03-07 18:16:05,674 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [4924927d] FINISH, SetVdsStatusVDSCommand, log id: 2a1f6742 >2013-03-07 18:16:05,680 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-80) [4924927d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 18:16:05,699 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-80) [4f713c07] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:16:05,702 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-80) [6cfcca05] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:16:05,703 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-80) [6cfcca05] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 18:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [57025725] Autorecovering 1 hosts >2013-03-07 18:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [57025725] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [6b472282] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [6b472282] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [6b472282] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3bbf4240 >2013-03-07 18:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [6b472282] FINISH, SetVdsStatusVDSCommand, log id: 3bbf4240 >2013-03-07 18:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-18) [6b472282] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 162d068b >2013-03-07 18:20:01,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [6b472282] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7073161 >2013-03-07 18:20:01,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [6b472282] FINISH, GetHardwareInfoVDSCommand, log id: 7073161 >2013-03-07 18:20:01,063 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [37499104] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:20:01,066 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [6d0c39b6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:20:01,067 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-18) [6d0c39b6] FINISH, ActivateVdsVDSCommand, log id: 162d068b >2013-03-07 18:20:01,087 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [6d0c39b6] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:20:01,088 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [6d0c39b6] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 18:20:01,090 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [6d0c39b6] Autorecovering 1 storage domains >2013-03-07 18:20:01,090 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [6d0c39b6] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 18:20:01,091 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [1b9ae193] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:20:01,091 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [1b9ae193] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 6:20 PM >2013-03-07 18:20:01,094 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [1b9ae193] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 6:20 PM >2013-03-07 18:20:02,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1db36a3a >2013-03-07 18:20:02,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) FINISH, GetHardwareInfoVDSCommand, log id: 1db36a3a >2013-03-07 18:20:02,967 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-25) [4faad971] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 18:20:02,971 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-25) [7b22ca04] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 18:20:02,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: f1676bd >2013-03-07 18:21:09,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 18:21:09,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:21:09,021 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 18:21:09,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: f1676bd >2013-03-07 18:21:09,025 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [6021f7a1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:21:09,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6021f7a1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3bd32664 >2013-03-07 18:21:09,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [6021f7a1] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 18:21:09,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6021f7a1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3bd32664 >2013-03-07 18:21:09,048 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [6021f7a1] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 18:21:09,048 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [7b22ca04] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 18:21:09,058 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-25) [7e9391d5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:21:09,059 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [7e9391d5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 56172328 >2013-03-07 18:21:09,061 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [7e9391d5] FINISH, SetVdsStatusVDSCommand, log id: 56172328 >2013-03-07 18:21:09,066 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-25) [7e9391d5] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 18:21:09,086 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-25) [500114be] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:21:09,088 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-25) [6c4867d5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:21:09,090 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-25) [6c4867d5] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 18:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [5e3d6e68] Autorecovering 1 hosts >2013-03-07 18:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [5e3d6e68] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-39) [5538dc28] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-39) [5538dc28] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [5538dc28] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 73578b7b >2013-03-07 18:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [5538dc28] FINISH, SetVdsStatusVDSCommand, log id: 73578b7b >2013-03-07 18:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-39) [5538dc28] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6bd6db24 >2013-03-07 18:25:00,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [5538dc28] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5ec33c0c >2013-03-07 18:25:00,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [5538dc28] FINISH, GetHardwareInfoVDSCommand, log id: 5ec33c0c >2013-03-07 18:25:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-39) [2970f248] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:25:00,095 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-39) [33fc3993] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:25:00,096 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-39) [33fc3993] FINISH, ActivateVdsVDSCommand, log id: 6bd6db24 >2013-03-07 18:25:00,116 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-39) [33fc3993] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:25:00,117 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-39) [33fc3993] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 18:25:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [33fc3993] Autorecovering 1 storage domains >2013-03-07 18:25:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [33fc3993] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 18:25:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-39) [63ed97da] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:25:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-39) [63ed97da] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 6:25 PM >2013-03-07 18:25:00,122 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-39) [63ed97da] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 6:25 PM >2013-03-07 18:25:00,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [1e67011c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3c42da9f >2013-03-07 18:25:00,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [1e67011c] FINISH, GetHardwareInfoVDSCommand, log id: 3c42da9f >2013-03-07 18:25:00,326 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-4) [4d73a595] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 18:25:00,330 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-4) [6a81f00d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 18:25:00,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3ee21479 >2013-03-07 18:26:05,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 18:26:05,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:26:05,346 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 18:26:05,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 3ee21479 >2013-03-07 18:26:05,349 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [fab4d51] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:26:05,367 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [fab4d51] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 768fd1ce >2013-03-07 18:26:05,367 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [fab4d51] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 18:26:05,367 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [fab4d51] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 768fd1ce >2013-03-07 18:26:05,369 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [fab4d51] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 18:26:05,370 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [6a81f00d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 18:26:05,383 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-4) [72b9cf7d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:26:05,385 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [72b9cf7d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 56b98143 >2013-03-07 18:26:05,386 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [72b9cf7d] FINISH, SetVdsStatusVDSCommand, log id: 56b98143 >2013-03-07 18:26:05,392 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-4) [72b9cf7d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 18:26:05,426 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-4) [30be2ca3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:26:05,428 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-4) [61ae1732] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:26:05,430 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-4) [61ae1732] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 18:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) Autorecovering 1 hosts >2013-03-07 18:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-11) [485e1d0c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-11) [485e1d0c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [485e1d0c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2f95379a >2013-03-07 18:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [485e1d0c] FINISH, SetVdsStatusVDSCommand, log id: 2f95379a >2013-03-07 18:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-11) [485e1d0c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2edae658 >2013-03-07 18:30:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [485e1d0c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2927d8ed >2013-03-07 18:30:00,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [485e1d0c] FINISH, GetHardwareInfoVDSCommand, log id: 2927d8ed >2013-03-07 18:30:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-11) [29cd78cc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:30:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-11) [7270e6ef] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:30:00,206 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-11) [7270e6ef] FINISH, ActivateVdsVDSCommand, log id: 2edae658 >2013-03-07 18:30:00,225 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-11) [7270e6ef] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:30:00,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-11) [7270e6ef] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 18:30:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) [7270e6ef] Autorecovering 1 storage domains >2013-03-07 18:30:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) [7270e6ef] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 18:30:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-11) [37e7b9cb] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:30:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-11) [37e7b9cb] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 6:30 PM >2013-03-07 18:30:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-11) [37e7b9cb] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 6:30 PM >2013-03-07 18:30:02,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 120041e3 >2013-03-07 18:30:02,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) FINISH, GetHardwareInfoVDSCommand, log id: 120041e3 >2013-03-07 18:30:02,675 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-10) [6899330e] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 18:30:02,679 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-10) [2317fe80] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 18:30:02,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4f3e7937 >2013-03-07 18:30:33,700 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-59) [eaad4a6] Start refreshing all users data >2013-03-07 18:30:34,563 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) [5f57ee5e] Attempting to update VMs/Templates Ovf. >2013-03-07 18:30:34,564 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) [5f57ee5e] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 18:30:34,571 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) [5f57ee5e] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 18:30:34,571 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) [5f57ee5e] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 18:30:34,572 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) [5f57ee5e] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 18:30:34,572 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) [5f57ee5e] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 18:30:34,575 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) [5f57ee5e] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 18:31:05,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 18:31:05,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:31:05,678 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 18:31:05,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 4f3e7937 >2013-03-07 18:31:05,681 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [35ac482a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:31:05,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [35ac482a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 8f53a68 >2013-03-07 18:31:05,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [35ac482a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 18:31:05,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [35ac482a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 8f53a68 >2013-03-07 18:31:05,699 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [35ac482a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 18:31:05,699 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [2317fe80] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 18:31:05,711 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-10) [2d964408] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:31:05,712 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [2d964408] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6c4a5867 >2013-03-07 18:31:05,714 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [2d964408] FINISH, SetVdsStatusVDSCommand, log id: 6c4a5867 >2013-03-07 18:31:05,720 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-10) [2d964408] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 18:31:05,745 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-10) [4e22fadd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:31:05,747 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-10) [12f5a35f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:31:05,750 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-10) [12f5a35f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 18:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) Autorecovering 1 hosts >2013-03-07 18:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [7a3565e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [7a3565e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [7a3565e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5c25522b >2013-03-07 18:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [7a3565e] FINISH, SetVdsStatusVDSCommand, log id: 5c25522b >2013-03-07 18:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-64) [7a3565e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 513f4b80 >2013-03-07 18:35:00,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [7a3565e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2aaee6e0 >2013-03-07 18:35:00,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [7a3565e] FINISH, GetHardwareInfoVDSCommand, log id: 2aaee6e0 >2013-03-07 18:35:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-64) [1598d1a8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:35:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-64) [3965db69] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:35:00,206 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-64) [3965db69] FINISH, ActivateVdsVDSCommand, log id: 513f4b80 >2013-03-07 18:35:00,226 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [3965db69] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:35:00,227 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-64) [3965db69] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 18:35:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [3965db69] Autorecovering 1 storage domains >2013-03-07 18:35:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [3965db69] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 18:35:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [3099d800] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:35:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [3099d800] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 6:35 PM >2013-03-07 18:35:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [3099d800] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 6:35 PM >2013-03-07 18:35:02,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [63ed97da] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2b8b276d >2013-03-07 18:35:02,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [63ed97da] FINISH, GetHardwareInfoVDSCommand, log id: 2b8b276d >2013-03-07 18:35:02,992 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-39) [5250290] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 18:35:02,996 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-39) [9672dbd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 18:35:03,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 17f0ff4e >2013-03-07 18:36:06,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 18:36:06,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:36:06,988 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 18:36:06,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 17f0ff4e >2013-03-07 18:36:06,992 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [1a0e0fd6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:36:07,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1a0e0fd6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 62ea6245 >2013-03-07 18:36:07,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [1a0e0fd6] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 18:36:07,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1a0e0fd6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 62ea6245 >2013-03-07 18:36:07,010 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [1a0e0fd6] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 18:36:07,010 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [9672dbd] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 18:36:07,021 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-39) [3ec459ac] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:36:07,022 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [3ec459ac] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7d7a45c7 >2013-03-07 18:36:07,024 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [3ec459ac] FINISH, SetVdsStatusVDSCommand, log id: 7d7a45c7 >2013-03-07 18:36:07,033 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-39) [3ec459ac] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 18:36:07,053 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-39) [618ab432] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:36:07,056 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-39) [6fe3b474] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:36:07,057 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-39) [6fe3b474] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 18:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [111f6844] Autorecovering 1 hosts >2013-03-07 18:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [111f6844] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [11f7a183] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [11f7a183] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [11f7a183] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 21ae7fdf >2013-03-07 18:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [11f7a183] FINISH, SetVdsStatusVDSCommand, log id: 21ae7fdf >2013-03-07 18:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-46) [11f7a183] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2948b6cc >2013-03-07 18:40:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [11f7a183] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: dc2b57a >2013-03-07 18:40:00,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [11f7a183] FINISH, GetHardwareInfoVDSCommand, log id: dc2b57a >2013-03-07 18:40:00,221 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-46) [2fb8b7c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:40:00,223 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-46) [69bc68e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:40:00,225 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-46) [69bc68e] FINISH, ActivateVdsVDSCommand, log id: 2948b6cc >2013-03-07 18:40:00,244 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [69bc68e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:40:00,245 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-46) [69bc68e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 18:40:00,247 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [69bc68e] Autorecovering 1 storage domains >2013-03-07 18:40:00,247 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [69bc68e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 18:40:00,248 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [28ad0e2a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:40:00,248 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [28ad0e2a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 6:40 PM >2013-03-07 18:40:00,251 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [28ad0e2a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 6:40 PM >2013-03-07 18:40:01,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [329a0098] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 324cbee6 >2013-03-07 18:40:01,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [329a0098] FINISH, GetHardwareInfoVDSCommand, log id: 324cbee6 >2013-03-07 18:40:01,292 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-8) [30bcd69d] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 18:40:01,296 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-8) [5b448481] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 18:40:01,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1e78418c >2013-03-07 18:41:06,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 18:41:06,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:41:06,290 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 18:41:06,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 1e78418c >2013-03-07 18:41:06,294 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [616425eb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:41:06,311 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [616425eb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4438d89b >2013-03-07 18:41:06,311 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [616425eb] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 18:41:06,311 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [616425eb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4438d89b >2013-03-07 18:41:06,313 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [616425eb] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 18:41:06,314 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [5b448481] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 18:41:06,324 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-8) [4b8f81df] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:41:06,325 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [4b8f81df] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 60321d2b >2013-03-07 18:41:06,326 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [4b8f81df] FINISH, SetVdsStatusVDSCommand, log id: 60321d2b >2013-03-07 18:41:06,333 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-8) [4b8f81df] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 18:41:06,353 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [52c97be6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:41:06,355 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [5487f815] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:41:06,357 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-8) [5487f815] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 18:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [5a14534c] Autorecovering 1 hosts >2013-03-07 18:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [5a14534c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:45:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [5e023a76] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [5e023a76] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [5e023a76] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 53f18ab5 >2013-03-07 18:45:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [5e023a76] FINISH, SetVdsStatusVDSCommand, log id: 53f18ab5 >2013-03-07 18:45:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [5e023a76] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4b325ee6 >2013-03-07 18:45:00,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [5e023a76] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 525d905f >2013-03-07 18:45:00,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [5e023a76] FINISH, GetHardwareInfoVDSCommand, log id: 525d905f >2013-03-07 18:45:00,226 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [68a9eaa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:45:00,229 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [28219308] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:45:00,230 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [28219308] FINISH, ActivateVdsVDSCommand, log id: 4b325ee6 >2013-03-07 18:45:00,250 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [28219308] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:45:00,251 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-94) [28219308] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 18:45:00,252 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [28219308] Autorecovering 1 storage domains >2013-03-07 18:45:00,253 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [28219308] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 18:45:00,253 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [113ae533] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:45:00,253 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [113ae533] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 6:45 PM >2013-03-07 18:45:00,256 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [113ae533] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 6:45 PM >2013-03-07 18:45:00,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 49e0ce40 >2013-03-07 18:45:00,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) FINISH, GetHardwareInfoVDSCommand, log id: 49e0ce40 >2013-03-07 18:45:00,606 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-56) [390a5db3] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 18:45:00,609 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-56) [7adda64f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 18:45:00,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 67ae1021 >2013-03-07 18:46:04,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 18:46:04,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:46:04,599 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 18:46:04,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 67ae1021 >2013-03-07 18:46:04,603 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [2ba59b4e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:46:04,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [2ba59b4e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 76b54ec >2013-03-07 18:46:04,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [2ba59b4e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 18:46:04,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [2ba59b4e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 76b54ec >2013-03-07 18:46:04,621 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [2ba59b4e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 18:46:04,621 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [7adda64f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 18:46:04,631 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-56) [66ed5501] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:46:04,633 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [66ed5501] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 12590d54 >2013-03-07 18:46:04,634 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [66ed5501] FINISH, SetVdsStatusVDSCommand, log id: 12590d54 >2013-03-07 18:46:04,641 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-56) [66ed5501] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 18:46:04,670 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-56) [1dbc700d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:46:04,672 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-56) [6c7f2de4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:46:04,673 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-56) [6c7f2de4] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 18:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [438bac47] Autorecovering 1 hosts >2013-03-07 18:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [438bac47] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [1e66fca] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [1e66fca] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [1e66fca] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2820d492 >2013-03-07 18:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [1e66fca] FINISH, SetVdsStatusVDSCommand, log id: 2820d492 >2013-03-07 18:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [1e66fca] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 32f1d7ab >2013-03-07 18:50:00,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [1e66fca] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 343dd36c >2013-03-07 18:50:00,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [1e66fca] FINISH, GetHardwareInfoVDSCommand, log id: 343dd36c >2013-03-07 18:50:00,263 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [2b550ac6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:50:00,265 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [53688377] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:50:00,267 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [53688377] FINISH, ActivateVdsVDSCommand, log id: 32f1d7ab >2013-03-07 18:50:00,286 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [53688377] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:50:00,287 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [53688377] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 18:50:00,289 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [53688377] Autorecovering 1 storage domains >2013-03-07 18:50:00,289 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [53688377] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 18:50:00,289 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [760cbcdf] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:50:00,290 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [760cbcdf] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 6:50 PM >2013-03-07 18:50:00,292 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [760cbcdf] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 6:50 PM >2013-03-07 18:50:01,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 63b75468 >2013-03-07 18:50:01,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) FINISH, GetHardwareInfoVDSCommand, log id: 63b75468 >2013-03-07 18:50:01,919 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-9) [97719c5] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 18:50:01,922 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-9) [36a2bd77] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 18:50:01,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 452ed47a >2013-03-07 18:51:05,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 18:51:05,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:51:05,952 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 18:51:05,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 452ed47a >2013-03-07 18:51:05,956 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [1487bbed] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:51:05,971 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1487bbed] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5ff1bcf4 >2013-03-07 18:51:05,971 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [1487bbed] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 18:51:05,971 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1487bbed] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5ff1bcf4 >2013-03-07 18:51:05,973 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [1487bbed] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 18:51:05,974 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [36a2bd77] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 18:51:05,984 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-9) [69da2f2b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:51:05,985 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [69da2f2b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 19009e74 >2013-03-07 18:51:05,986 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [69da2f2b] FINISH, SetVdsStatusVDSCommand, log id: 19009e74 >2013-03-07 18:51:05,993 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-9) [69da2f2b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 18:51:06,014 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-9) [38f35d41] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:51:06,017 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-9) [62bf8532] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:51:06,018 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-9) [62bf8532] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 18:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) Autorecovering 1 hosts >2013-03-07 18:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-49) [18febb31] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-49) [18febb31] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [18febb31] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7c92ea40 >2013-03-07 18:55:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [18febb31] FINISH, SetVdsStatusVDSCommand, log id: 7c92ea40 >2013-03-07 18:55:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-49) [18febb31] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 344f5775 >2013-03-07 18:55:02,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [18febb31] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4f8bf8be >2013-03-07 18:55:02,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [18febb31] FINISH, GetHardwareInfoVDSCommand, log id: 4f8bf8be >2013-03-07 18:55:02,539 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-49) [604a430d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:55:02,542 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-49) [746aa338] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:55:02,543 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-49) [746aa338] FINISH, ActivateVdsVDSCommand, log id: 344f5775 >2013-03-07 18:55:02,563 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-49) [746aa338] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 18:55:02,564 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-49) [746aa338] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 18:55:02,566 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [746aa338] Autorecovering 1 storage domains >2013-03-07 18:55:02,567 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [746aa338] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 18:55:02,567 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-49) [68082913] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:55:02,568 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-49) [68082913] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 6:55 PM >2013-03-07 18:55:02,570 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-49) [68082913] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 6:55 PM >2013-03-07 18:55:02,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [4fdb94a0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5d2ffffc >2013-03-07 18:55:02,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [4fdb94a0] FINISH, GetHardwareInfoVDSCommand, log id: 5d2ffffc >2013-03-07 18:55:02,647 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-87) [71d29a7a] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 18:55:02,651 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-87) [26c09864] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 18:55:02,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6ce4c8ca >2013-03-07 18:56:06,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 18:56:06,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 18:56:06,662 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 18:56:06,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 6ce4c8ca >2013-03-07 18:56:06,666 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [5ac1a00f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 18:56:06,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5ac1a00f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 73221626 >2013-03-07 18:56:06,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [5ac1a00f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 18:56:06,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5ac1a00f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 73221626 >2013-03-07 18:56:06,685 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [5ac1a00f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 18:56:06,685 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [26c09864] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 18:56:06,696 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-87) [2ca0bc72] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:56:06,698 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [2ca0bc72] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 30aff587 >2013-03-07 18:56:06,699 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [2ca0bc72] FINISH, SetVdsStatusVDSCommand, log id: 30aff587 >2013-03-07 18:56:06,707 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [2ca0bc72] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 18:56:06,741 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [7a17c289] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:56:06,744 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [5ac3919a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 18:56:06,745 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-87) [5ac3919a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 19:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) Autorecovering 1 hosts >2013-03-07 19:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-98) [5096a0bf] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-98) [5096a0bf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-98) [5096a0bf] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5dddea1c >2013-03-07 19:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-98) [5096a0bf] FINISH, SetVdsStatusVDSCommand, log id: 5dddea1c >2013-03-07 19:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-98) [5096a0bf] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 33fc9841 >2013-03-07 19:00:00,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-98) [5096a0bf] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 16f5021f >2013-03-07 19:00:00,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-98) [5096a0bf] FINISH, GetHardwareInfoVDSCommand, log id: 16f5021f >2013-03-07 19:00:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-98) [4899ab1e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:00:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-98) [2ddcbc5a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:00:00,095 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-98) [2ddcbc5a] FINISH, ActivateVdsVDSCommand, log id: 33fc9841 >2013-03-07 19:00:00,115 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-98) [2ddcbc5a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:00:00,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-98) [2ddcbc5a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 19:00:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [2ddcbc5a] Autorecovering 1 storage domains >2013-03-07 19:00:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [2ddcbc5a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 19:00:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-98) [19a5f55] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:00:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-98) [19a5f55] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 7:00 PM >2013-03-07 19:00:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-98) [19a5f55] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 7:00 PM >2013-03-07 19:00:00,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [1589ff66] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 10903ba0 >2013-03-07 19:00:00,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [1589ff66] FINISH, GetHardwareInfoVDSCommand, log id: 10903ba0 >2013-03-07 19:00:00,991 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-70) [569538d7] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 19:00:00,995 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-70) [3ad1bcd1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 19:00:01,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 442b3c10 >2013-03-07 19:01:04,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 19:01:04,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:01:04,985 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 19:01:04,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 442b3c10 >2013-03-07 19:01:04,989 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [62747f6d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:01:05,012 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [62747f6d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2214ca8d >2013-03-07 19:01:05,012 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [62747f6d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 19:01:05,012 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [62747f6d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2214ca8d >2013-03-07 19:01:05,014 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [62747f6d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 19:01:05,015 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [3ad1bcd1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 19:01:05,024 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-70) [173efe25] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:01:05,026 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [173efe25] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 33f67de2 >2013-03-07 19:01:05,027 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [173efe25] FINISH, SetVdsStatusVDSCommand, log id: 33f67de2 >2013-03-07 19:01:05,032 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-70) [173efe25] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 19:01:05,051 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-70) [210d7bc7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:01:05,053 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-70) [553221d9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:01:05,055 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-70) [553221d9] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 19:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [2466e97f] Autorecovering 1 hosts >2013-03-07 19:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [2466e97f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-82) [60877c4e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-82) [60877c4e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [60877c4e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 61ed23e9 >2013-03-07 19:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [60877c4e] FINISH, SetVdsStatusVDSCommand, log id: 61ed23e9 >2013-03-07 19:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-82) [60877c4e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 432e3e46 >2013-03-07 19:05:00,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [60877c4e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 221b2d65 >2013-03-07 19:05:00,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [60877c4e] FINISH, GetHardwareInfoVDSCommand, log id: 221b2d65 >2013-03-07 19:05:00,408 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-82) [69f0025a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:05:00,411 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-82) [1a417655] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:05:00,412 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-82) [1a417655] FINISH, ActivateVdsVDSCommand, log id: 432e3e46 >2013-03-07 19:05:00,432 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-82) [1a417655] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:05:00,433 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-82) [1a417655] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 19:05:00,434 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [1a417655] Autorecovering 1 storage domains >2013-03-07 19:05:00,434 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [1a417655] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 19:05:00,435 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-82) [3a13c4c1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:05:00,435 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-82) [3a13c4c1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 7:05 PM >2013-03-07 19:05:00,438 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-82) [3a13c4c1] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 7:05 PM >2013-03-07 19:05:02,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [6c811f72] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4425a105 >2013-03-07 19:05:02,267 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [6c811f72] FINISH, GetHardwareInfoVDSCommand, log id: 4425a105 >2013-03-07 19:05:02,305 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-43) [21f904d3] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 19:05:02,308 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-43) [5514aee2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 19:05:02,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4bee39cb >2013-03-07 19:06:06,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 19:06:06,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:06:06,313 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 19:06:06,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 4bee39cb >2013-03-07 19:06:06,316 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [6f3daa5d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:06:06,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6f3daa5d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 641138c4 >2013-03-07 19:06:06,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [6f3daa5d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 19:06:06,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6f3daa5d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 641138c4 >2013-03-07 19:06:06,342 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [5514aee2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 19:06:06,353 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-43) [623ed44a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:06:06,354 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [623ed44a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 21883f2c >2013-03-07 19:06:06,356 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [623ed44a] FINISH, SetVdsStatusVDSCommand, log id: 21883f2c >2013-03-07 19:06:06,363 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [623ed44a] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 19:06:06,390 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [4f0fe566] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:06:06,393 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [66409cf7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:06:06,398 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-43) [66409cf7] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 19:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) Autorecovering 1 hosts >2013-03-07 19:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [1f46b79b] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [1f46b79b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [1f46b79b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 722f4e7b >2013-03-07 19:10:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [1f46b79b] FINISH, SetVdsStatusVDSCommand, log id: 722f4e7b >2013-03-07 19:10:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-89) [1f46b79b] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1af60545 >2013-03-07 19:10:00,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [1f46b79b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 69450f25 >2013-03-07 19:10:00,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [1f46b79b] FINISH, GetHardwareInfoVDSCommand, log id: 69450f25 >2013-03-07 19:10:00,095 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-89) [1aaf2f65] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:10:00,097 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-89) [7ac852e9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:10:00,099 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-89) [7ac852e9] FINISH, ActivateVdsVDSCommand, log id: 1af60545 >2013-03-07 19:10:00,118 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [7ac852e9] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:10:00,119 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-89) [7ac852e9] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 19:10:00,120 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [7ac852e9] Autorecovering 1 storage domains >2013-03-07 19:10:00,120 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [7ac852e9] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 19:10:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [7f8f1391] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:10:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [7f8f1391] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 7:10 PM >2013-03-07 19:10:00,124 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [7f8f1391] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 7:10 PM >2013-03-07 19:10:00,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 87a0366 >2013-03-07 19:10:00,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) FINISH, GetHardwareInfoVDSCommand, log id: 87a0366 >2013-03-07 19:10:00,646 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-93) [69da32b1] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 19:10:00,650 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-93) [22cc3a9a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 19:10:00,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 66f5e56b >2013-03-07 19:11:04,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 19:11:04,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:11:04,647 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 19:11:04,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 66f5e56b >2013-03-07 19:11:04,651 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [31296417] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:11:04,666 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [31296417] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3c5e95f4 >2013-03-07 19:11:04,666 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [31296417] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 19:11:04,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [31296417] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3c5e95f4 >2013-03-07 19:11:04,669 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [31296417] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 19:11:04,669 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [22cc3a9a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 19:11:04,680 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-93) [20d14eb8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:11:04,681 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [20d14eb8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1b3e15e6 >2013-03-07 19:11:04,682 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [20d14eb8] FINISH, SetVdsStatusVDSCommand, log id: 1b3e15e6 >2013-03-07 19:11:04,688 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-93) [20d14eb8] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 19:11:04,709 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [3d560530] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:11:04,712 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [55e61d48] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:11:04,713 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-93) [55e61d48] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 19:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) Autorecovering 1 hosts >2013-03-07 19:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:15:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [74ba0a5f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [74ba0a5f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [74ba0a5f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 48da8d39 >2013-03-07 19:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [74ba0a5f] FINISH, SetVdsStatusVDSCommand, log id: 48da8d39 >2013-03-07 19:15:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-35) [74ba0a5f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 51309d37 >2013-03-07 19:15:00,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [74ba0a5f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1e63a41 >2013-03-07 19:15:00,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [74ba0a5f] FINISH, GetHardwareInfoVDSCommand, log id: 1e63a41 >2013-03-07 19:15:00,461 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-35) [4bf42956] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:15:00,463 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-35) [49b28454] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:15:00,465 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-35) [49b28454] FINISH, ActivateVdsVDSCommand, log id: 51309d37 >2013-03-07 19:15:00,485 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [49b28454] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:15:00,486 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-35) [49b28454] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 19:15:00,488 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [49b28454] Autorecovering 1 storage domains >2013-03-07 19:15:00,488 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [49b28454] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 19:15:00,488 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [f7c1140] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:15:00,489 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [f7c1140] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 7:15 PM >2013-03-07 19:15:00,491 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [f7c1140] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 7:15 PM >2013-03-07 19:15:01,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 639633e0 >2013-03-07 19:15:01,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) FINISH, GetHardwareInfoVDSCommand, log id: 639633e0 >2013-03-07 19:15:01,948 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-33) [5e366b84] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 19:15:01,951 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-33) [18648f5d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 19:15:01,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4c82d843 >2013-03-07 19:16:04,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 19:16:04,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:16:04,938 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 19:16:04,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 4c82d843 >2013-03-07 19:16:04,942 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [4e591b82] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:16:04,956 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [4e591b82] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2671cadd >2013-03-07 19:16:04,957 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [4e591b82] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 19:16:04,957 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [4e591b82] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2671cadd >2013-03-07 19:16:04,959 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [4e591b82] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 19:16:04,959 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [18648f5d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 19:16:04,969 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-33) [7a7af153] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:16:04,974 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [7a7af153] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 46fea1d5 >2013-03-07 19:16:04,975 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [7a7af153] FINISH, SetVdsStatusVDSCommand, log id: 46fea1d5 >2013-03-07 19:16:04,980 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-33) [7a7af153] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 19:16:05,000 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-33) [183db03] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:16:05,003 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-33) [6ad3d54c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:16:05,004 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-33) [6ad3d54c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 19:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [7c43979f] Autorecovering 1 hosts >2013-03-07 19:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [7c43979f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [161df4bd] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [161df4bd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [161df4bd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5691da9c >2013-03-07 19:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [161df4bd] FINISH, SetVdsStatusVDSCommand, log id: 5691da9c >2013-03-07 19:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [161df4bd] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 693248ac >2013-03-07 19:20:00,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [161df4bd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6aab35a >2013-03-07 19:20:00,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [161df4bd] FINISH, GetHardwareInfoVDSCommand, log id: 6aab35a >2013-03-07 19:20:00,106 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-79) [6ca31c5f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:20:00,108 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-79) [7af3d458] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:20:00,110 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [7af3d458] FINISH, ActivateVdsVDSCommand, log id: 693248ac >2013-03-07 19:20:00,129 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [7af3d458] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:20:00,130 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-79) [7af3d458] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 19:20:00,131 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [7af3d458] Autorecovering 1 storage domains >2013-03-07 19:20:00,131 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [7af3d458] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 19:20:00,132 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [c6fb43f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:20:00,132 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [c6fb43f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 7:20 PM >2013-03-07 19:20:00,135 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [c6fb43f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 7:20 PM >2013-03-07 19:20:02,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-23) [7d35bcf3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3338a475 >2013-03-07 19:20:02,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-23) [7d35bcf3] FINISH, GetHardwareInfoVDSCommand, log id: 3338a475 >2013-03-07 19:20:02,249 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-23) [1be6fbad] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 19:20:02,253 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-23) [18f76326] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 19:20:02,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 49c603e5 >2013-03-07 19:21:07,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 19:21:07,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:21:07,269 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 19:21:07,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 49c603e5 >2013-03-07 19:21:07,273 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [1a497c86] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:21:07,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [1a497c86] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 360496c2 >2013-03-07 19:21:07,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [1a497c86] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 19:21:07,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [1a497c86] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 360496c2 >2013-03-07 19:21:07,291 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [1a497c86] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 19:21:07,291 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [18f76326] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 19:21:07,302 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-23) [13a07f65] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:21:07,303 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-23) [13a07f65] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 46861dcb >2013-03-07 19:21:07,305 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-23) [13a07f65] FINISH, SetVdsStatusVDSCommand, log id: 46861dcb >2013-03-07 19:21:07,313 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-23) [13a07f65] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 19:21:07,334 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-23) [45345d7d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:21:07,340 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-23) [2a5d255b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:21:07,341 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-23) [2a5d255b] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 19:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) [182032a1] Autorecovering 1 hosts >2013-03-07 19:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) [182032a1] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-34) [66e7e888] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-34) [66e7e888] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [66e7e888] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 63c481db >2013-03-07 19:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [66e7e888] FINISH, SetVdsStatusVDSCommand, log id: 63c481db >2013-03-07 19:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-34) [66e7e888] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 659de083 >2013-03-07 19:25:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [66e7e888] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5b388a4 >2013-03-07 19:25:00,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [66e7e888] FINISH, GetHardwareInfoVDSCommand, log id: 5b388a4 >2013-03-07 19:25:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-34) [479a261d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:25:00,208 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-34) [3c1748cb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:25:00,209 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-34) [3c1748cb] FINISH, ActivateVdsVDSCommand, log id: 659de083 >2013-03-07 19:25:00,229 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-34) [3c1748cb] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:25:00,230 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-34) [3c1748cb] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 19:25:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) [3c1748cb] Autorecovering 1 storage domains >2013-03-07 19:25:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) [3c1748cb] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 19:25:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-34) [27e3fb5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:25:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-34) [27e3fb5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 7:25 PM >2013-03-07 19:25:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-34) [27e3fb5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 7:25 PM >2013-03-07 19:25:01,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 64fa317a >2013-03-07 19:25:01,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) FINISH, GetHardwareInfoVDSCommand, log id: 64fa317a >2013-03-07 19:25:01,587 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-32) [f618b14] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 19:25:01,591 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-32) [5d11ff28] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 19:25:01,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 35196b19 >2013-03-07 19:26:05,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 19:26:05,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:26:05,605 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 19:26:05,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 35196b19 >2013-03-07 19:26:05,609 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [2ce0515f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:26:05,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [2ce0515f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 60b6c712 >2013-03-07 19:26:05,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [2ce0515f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 19:26:05,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [2ce0515f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 60b6c712 >2013-03-07 19:26:05,626 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [2ce0515f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 19:26:05,627 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [5d11ff28] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 19:26:05,637 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-32) [6a1aceb0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:26:05,638 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [6a1aceb0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 53e42b96 >2013-03-07 19:26:05,640 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [6a1aceb0] FINISH, SetVdsStatusVDSCommand, log id: 53e42b96 >2013-03-07 19:26:05,646 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-32) [6a1aceb0] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 19:26:05,666 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-32) [4a9330f4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:26:05,669 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-32) [53b9b3d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:26:05,670 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-32) [53b9b3d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 19:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [6f2114a2] Autorecovering 1 hosts >2013-03-07 19:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [6f2114a2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [25a887a5] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [25a887a5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [25a887a5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 33812ef8 >2013-03-07 19:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [25a887a5] FINISH, SetVdsStatusVDSCommand, log id: 33812ef8 >2013-03-07 19:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [25a887a5] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 29cb8081 >2013-03-07 19:30:00,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [25a887a5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5db553d1 >2013-03-07 19:30:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [25a887a5] FINISH, GetHardwareInfoVDSCommand, log id: 5db553d1 >2013-03-07 19:30:00,184 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [7bf7db3e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:30:00,186 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [2e289d73] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:30:00,187 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [2e289d73] FINISH, ActivateVdsVDSCommand, log id: 29cb8081 >2013-03-07 19:30:00,207 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [2e289d73] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:30:00,208 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [2e289d73] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 19:30:00,209 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [2e289d73] Autorecovering 1 storage domains >2013-03-07 19:30:00,209 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [2e289d73] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 19:30:00,210 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [9cd5977] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:30:00,210 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [9cd5977] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 7:30 PM >2013-03-07 19:30:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [9cd5977] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 7:30 PM >2013-03-07 19:30:02,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 288d16be >2013-03-07 19:30:02,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) FINISH, GetHardwareInfoVDSCommand, log id: 288d16be >2013-03-07 19:30:02,918 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-88) [79c6abe7] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 19:30:02,922 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-88) [49440e5d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 19:30:02,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 77b8cac7 >2013-03-07 19:30:33,702 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-43) [66409cf7] Start refreshing all users data >2013-03-07 19:30:34,576 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-11) [37e7b9cb] Attempting to update VMs/Templates Ovf. >2013-03-07 19:30:34,577 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-11) [37e7b9cb] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 19:30:34,584 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-11) [37e7b9cb] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 19:30:34,584 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-11) [37e7b9cb] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 19:30:34,587 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-11) [37e7b9cb] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 19:30:34,587 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-11) [37e7b9cb] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 19:30:34,588 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-11) [37e7b9cb] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 19:31:06,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 19:31:06,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:31:06,915 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 19:31:06,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 77b8cac7 >2013-03-07 19:31:06,918 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [696d0474] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:31:06,933 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [696d0474] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 795472f8 >2013-03-07 19:31:06,934 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [696d0474] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 19:31:06,934 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [696d0474] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 795472f8 >2013-03-07 19:31:06,936 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [696d0474] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 19:31:06,936 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [49440e5d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 19:31:06,947 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-88) [3fbe49c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:31:06,948 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [3fbe49c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1181a4db >2013-03-07 19:31:06,949 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [3fbe49c] FINISH, SetVdsStatusVDSCommand, log id: 1181a4db >2013-03-07 19:31:06,955 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-88) [3fbe49c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 19:31:06,978 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-88) [1489d5e2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:31:06,980 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-88) [4608f341] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:31:06,982 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-88) [4608f341] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 19:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [7c6dabac] Autorecovering 1 hosts >2013-03-07 19:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [7c6dabac] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [4d95db7e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [4d95db7e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [4d95db7e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4f640fe8 >2013-03-07 19:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [4d95db7e] FINISH, SetVdsStatusVDSCommand, log id: 4f640fe8 >2013-03-07 19:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-24) [4d95db7e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2300ea36 >2013-03-07 19:35:00,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [4d95db7e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2d60bc36 >2013-03-07 19:35:00,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [4d95db7e] FINISH, GetHardwareInfoVDSCommand, log id: 2d60bc36 >2013-03-07 19:35:00,457 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-24) [5d5512bd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:35:00,459 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-24) [5367f7c8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:35:00,461 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-24) [5367f7c8] FINISH, ActivateVdsVDSCommand, log id: 2300ea36 >2013-03-07 19:35:00,480 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [5367f7c8] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:35:00,481 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-24) [5367f7c8] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 19:35:00,482 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [5367f7c8] Autorecovering 1 storage domains >2013-03-07 19:35:00,482 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [5367f7c8] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 19:35:00,483 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [5812b250] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:35:00,483 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [5812b250] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 7:35 PM >2013-03-07 19:35:00,486 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [5812b250] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 7:35 PM >2013-03-07 19:35:01,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-71) [1b4c1700] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 288eea94 >2013-03-07 19:35:01,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-71) [1b4c1700] FINISH, GetHardwareInfoVDSCommand, log id: 288eea94 >2013-03-07 19:35:01,221 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-71) [511adc3e] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 19:35:01,224 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-71) [a7dc758] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 19:35:01,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2aa80dda >2013-03-07 19:36:08,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 19:36:08,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:36:08,221 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 19:36:08,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2aa80dda >2013-03-07 19:36:08,225 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [36740013] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:36:08,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [36740013] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 463e202 >2013-03-07 19:36:08,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [36740013] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 19:36:08,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [36740013] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 463e202 >2013-03-07 19:36:08,245 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [36740013] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 19:36:08,246 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [a7dc758] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 19:36:08,255 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-71) [312f6eb8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:36:08,257 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-71) [312f6eb8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5cbeeb62 >2013-03-07 19:36:08,258 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-71) [312f6eb8] FINISH, SetVdsStatusVDSCommand, log id: 5cbeeb62 >2013-03-07 19:36:08,264 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-71) [312f6eb8] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 19:36:08,284 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-71) [79924d32] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:36:08,286 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-71) [4d98a416] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:36:08,288 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-71) [4d98a416] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 19:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [2c941578] Autorecovering 1 hosts >2013-03-07 19:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [2c941578] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:40:00,020 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [56a77de9] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:40:00,020 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [56a77de9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:40:00,024 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [56a77de9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 257e0dc4 >2013-03-07 19:40:00,025 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [56a77de9] FINISH, SetVdsStatusVDSCommand, log id: 257e0dc4 >2013-03-07 19:40:00,028 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [56a77de9] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 60bb04a3 >2013-03-07 19:40:00,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [56a77de9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3159407e >2013-03-07 19:40:00,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [56a77de9] FINISH, GetHardwareInfoVDSCommand, log id: 3159407e >2013-03-07 19:40:00,107 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-14) [cb9ae64] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:40:00,110 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-14) [44c871e4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:40:00,111 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [44c871e4] FINISH, ActivateVdsVDSCommand, log id: 60bb04a3 >2013-03-07 19:40:00,131 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [44c871e4] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:40:00,132 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-14) [44c871e4] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 19:40:00,133 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [44c871e4] Autorecovering 1 storage domains >2013-03-07 19:40:00,133 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [44c871e4] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 19:40:00,134 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [77d00afb] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:40:00,134 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [77d00afb] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 7:40 PM >2013-03-07 19:40:00,137 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [77d00afb] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 7:40 PM >2013-03-07 19:40:02,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [5f9d4414] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 73591df6 >2013-03-07 19:40:02,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [5f9d4414] FINISH, GetHardwareInfoVDSCommand, log id: 73591df6 >2013-03-07 19:40:02,539 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-77) [792bcdc2] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 19:40:02,542 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-77) [7f79f7dc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 19:40:02,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: aaeb4b2 >2013-03-07 19:41:07,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 19:41:07,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:41:07,536 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 19:41:07,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: aaeb4b2 >2013-03-07 19:41:07,541 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [36af7aa4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:41:07,555 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [36af7aa4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 23d0655b >2013-03-07 19:41:07,556 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [36af7aa4] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 19:41:07,556 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [36af7aa4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 23d0655b >2013-03-07 19:41:07,558 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [36af7aa4] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 19:41:07,558 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [7f79f7dc] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 19:41:07,571 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-77) [426a2984] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:41:07,573 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [426a2984] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 23bd4fee >2013-03-07 19:41:07,574 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [426a2984] FINISH, SetVdsStatusVDSCommand, log id: 23bd4fee >2013-03-07 19:41:07,580 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-77) [426a2984] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 19:41:07,617 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-77) [3daef73c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:41:07,620 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-77) [25cefe9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:41:07,621 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-77) [25cefe9] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 19:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) Autorecovering 1 hosts >2013-03-07 19:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [185274b5] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [185274b5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [185274b5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7b0b86bb >2013-03-07 19:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [185274b5] FINISH, SetVdsStatusVDSCommand, log id: 7b0b86bb >2013-03-07 19:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-27) [185274b5] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3010d8fc >2013-03-07 19:45:00,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [185274b5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 66b94a80 >2013-03-07 19:45:00,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [185274b5] FINISH, GetHardwareInfoVDSCommand, log id: 66b94a80 >2013-03-07 19:45:00,241 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-27) [793e52f9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:45:00,244 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-27) [60ba83aa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:45:00,245 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-27) [60ba83aa] FINISH, ActivateVdsVDSCommand, log id: 3010d8fc >2013-03-07 19:45:00,265 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [60ba83aa] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:45:00,266 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-27) [60ba83aa] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 19:45:00,267 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [60ba83aa] Autorecovering 1 storage domains >2013-03-07 19:45:00,267 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [60ba83aa] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 19:45:00,268 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [4ffad86] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:45:00,268 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [4ffad86] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 7:45 PM >2013-03-07 19:45:00,271 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [4ffad86] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 7:45 PM >2013-03-07 19:45:01,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 59dc65a2 >2013-03-07 19:45:01,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) FINISH, GetHardwareInfoVDSCommand, log id: 59dc65a2 >2013-03-07 19:45:01,868 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-20) [132844b] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 19:45:01,871 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-20) [345bbb2e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 19:45:01,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 223aaee4 >2013-03-07 19:46:06,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 19:46:06,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:46:06,170 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 19:46:06,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 223aaee4 >2013-03-07 19:46:06,174 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [4b0627a1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:46:06,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [4b0627a1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6a2b59eb >2013-03-07 19:46:06,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [4b0627a1] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 19:46:06,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [4b0627a1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6a2b59eb >2013-03-07 19:46:06,191 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [4b0627a1] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 19:46:06,192 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [345bbb2e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 19:46:06,202 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-20) [7cc35585] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:46:06,203 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [7cc35585] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 166ddfcd >2013-03-07 19:46:06,205 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [7cc35585] FINISH, SetVdsStatusVDSCommand, log id: 166ddfcd >2013-03-07 19:46:06,211 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-20) [7cc35585] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 19:46:06,230 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-20) [9134e2d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:46:06,233 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-20) [ffe013d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:46:06,234 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-20) [ffe013d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 19:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [62bf8532] Autorecovering 1 hosts >2013-03-07 19:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [62bf8532] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [7cb8bc] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [7cb8bc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [7cb8bc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3cd487d6 >2013-03-07 19:50:00,039 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [7cb8bc] FINISH, SetVdsStatusVDSCommand, log id: 3cd487d6 >2013-03-07 19:50:00,042 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-9) [7cb8bc] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1d3d7f55 >2013-03-07 19:50:00,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [7cb8bc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5bf774ae >2013-03-07 19:50:00,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [7cb8bc] FINISH, GetHardwareInfoVDSCommand, log id: 5bf774ae >2013-03-07 19:50:00,341 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-9) [59f5ea55] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:50:00,343 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-9) [5d4433ee] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:50:00,345 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-9) [5d4433ee] FINISH, ActivateVdsVDSCommand, log id: 1d3d7f55 >2013-03-07 19:50:00,364 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [5d4433ee] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:50:00,365 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-9) [5d4433ee] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 19:50:00,366 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [5d4433ee] Autorecovering 1 storage domains >2013-03-07 19:50:00,366 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [5d4433ee] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 19:50:00,367 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [27036916] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:50:00,367 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [27036916] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 7:50 PM >2013-03-07 19:50:00,370 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [27036916] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 7:50 PM >2013-03-07 19:50:00,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 170277e7 >2013-03-07 19:50:00,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) FINISH, GetHardwareInfoVDSCommand, log id: 170277e7 >2013-03-07 19:50:00,484 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-78) [5cf4b47c] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 19:50:00,488 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-78) [526a3527] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 19:50:00,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 74343249 >2013-03-07 19:51:06,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 19:51:06,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:51:06,529 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 19:51:06,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 74343249 >2013-03-07 19:51:06,533 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [70d4f966] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:51:06,548 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [70d4f966] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c4056ec >2013-03-07 19:51:06,548 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [70d4f966] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 19:51:06,548 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [70d4f966] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c4056ec >2013-03-07 19:51:06,551 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [70d4f966] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 19:51:06,551 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [526a3527] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 19:51:06,560 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-78) [7fe4bcec] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:51:06,562 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [7fe4bcec] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3c3bab58 >2013-03-07 19:51:06,563 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [7fe4bcec] FINISH, SetVdsStatusVDSCommand, log id: 3c3bab58 >2013-03-07 19:51:06,569 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-78) [7fe4bcec] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 19:51:06,589 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-78) [5d716639] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:51:06,592 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-78) [3f8dc449] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:51:06,593 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-78) [3f8dc449] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 19:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [5ac3919a] Autorecovering 1 hosts >2013-03-07 19:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [5ac3919a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [cd3912a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [cd3912a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [cd3912a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4da54391 >2013-03-07 19:55:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [cd3912a] FINISH, SetVdsStatusVDSCommand, log id: 4da54391 >2013-03-07 19:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-87) [cd3912a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4197e1cb >2013-03-07 19:55:00,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [cd3912a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 28074c4e >2013-03-07 19:55:00,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [cd3912a] FINISH, GetHardwareInfoVDSCommand, log id: 28074c4e >2013-03-07 19:55:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [62fd7d7b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:55:00,096 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [4b319e86] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:55:00,097 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-87) [4b319e86] FINISH, ActivateVdsVDSCommand, log id: 4197e1cb >2013-03-07 19:55:00,117 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [4b319e86] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 19:55:00,118 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [4b319e86] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 19:55:00,119 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [4b319e86] Autorecovering 1 storage domains >2013-03-07 19:55:00,119 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [4b319e86] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 19:55:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [34c22d61] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:55:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [34c22d61] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 7:55 PM >2013-03-07 19:55:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [34c22d61] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 7:55 PM >2013-03-07 19:55:00,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2921af81 >2013-03-07 19:55:00,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) FINISH, GetHardwareInfoVDSCommand, log id: 2921af81 >2013-03-07 19:55:00,834 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-19) [24f22d97] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 19:55:00,837 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-19) [112428d6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 19:55:00,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5dec6e6f >2013-03-07 19:56:04,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 19:56:04,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 19:56:04,831 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 19:56:04,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 5dec6e6f >2013-03-07 19:56:04,835 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [72bb775] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 19:56:04,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [72bb775] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 17ab35f5 >2013-03-07 19:56:04,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [72bb775] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 19:56:04,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [72bb775] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 17ab35f5 >2013-03-07 19:56:04,853 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [72bb775] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 19:56:04,854 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [112428d6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 19:56:04,863 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-19) [39ce2a9e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:56:04,865 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [39ce2a9e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 45cd34d0 >2013-03-07 19:56:04,866 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [39ce2a9e] FINISH, SetVdsStatusVDSCommand, log id: 45cd34d0 >2013-03-07 19:56:04,873 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-19) [39ce2a9e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 19:56:04,892 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-19) [24cfaaed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:56:04,895 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-19) [4d9a0ab0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 19:56:04,897 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-19) [4d9a0ab0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 20:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) Autorecovering 1 hosts >2013-03-07 20:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-61) [2af134c7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-61) [2af134c7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [2af134c7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2c2ecb3c >2013-03-07 20:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [2af134c7] FINISH, SetVdsStatusVDSCommand, log id: 2c2ecb3c >2013-03-07 20:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-61) [2af134c7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5917960b >2013-03-07 20:00:00,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [2af134c7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 54ec7070 >2013-03-07 20:00:00,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [2af134c7] FINISH, GetHardwareInfoVDSCommand, log id: 54ec7070 >2013-03-07 20:00:00,206 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-61) [e34f7d1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:00:00,208 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-61) [7ed222a5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:00:00,210 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-61) [7ed222a5] FINISH, ActivateVdsVDSCommand, log id: 5917960b >2013-03-07 20:00:00,229 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-61) [7ed222a5] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:00:00,230 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-61) [7ed222a5] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 20:00:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) [7ed222a5] Autorecovering 1 storage domains >2013-03-07 20:00:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) [7ed222a5] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 20:00:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-61) [4838c2ca] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:00:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-61) [4838c2ca] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 8:00 PM >2013-03-07 20:00:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-61) [4838c2ca] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 8:00 PM >2013-03-07 20:00:02,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [136fea9d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4d280b5 >2013-03-07 20:00:02,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [136fea9d] FINISH, GetHardwareInfoVDSCommand, log id: 4d280b5 >2013-03-07 20:00:02,148 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-67) [918c5c0] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 20:00:02,151 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-67) [66e10916] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 20:00:02,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 116abe00 >2013-03-07 20:01:08,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 20:01:08,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:01:08,160 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 20:01:08,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 116abe00 >2013-03-07 20:01:08,164 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [110d782f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:01:08,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [110d782f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5a4c7a18 >2013-03-07 20:01:08,190 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [110d782f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 20:01:08,190 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [110d782f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5a4c7a18 >2013-03-07 20:01:08,192 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [110d782f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 20:01:08,192 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [66e10916] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 20:01:08,202 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-67) [367dd20b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:01:08,204 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [367dd20b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6401bb5f >2013-03-07 20:01:08,205 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [367dd20b] FINISH, SetVdsStatusVDSCommand, log id: 6401bb5f >2013-03-07 20:01:08,211 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-67) [367dd20b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 20:01:08,231 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-67) [5dd93963] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:01:08,233 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-67) [419059ec] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:01:08,235 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-67) [419059ec] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 20:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [66409cf7] Autorecovering 1 hosts >2013-03-07 20:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [66409cf7] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [65fc28c7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [65fc28c7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [65fc28c7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2bdbe80d >2013-03-07 20:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [65fc28c7] FINISH, SetVdsStatusVDSCommand, log id: 2bdbe80d >2013-03-07 20:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [65fc28c7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 397c9334 >2013-03-07 20:05:00,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [65fc28c7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 789a73a3 >2013-03-07 20:05:00,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [65fc28c7] FINISH, GetHardwareInfoVDSCommand, log id: 789a73a3 >2013-03-07 20:05:00,244 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [7c7a5a6d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:05:00,246 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [1951c47] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:05:00,248 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [1951c47] FINISH, ActivateVdsVDSCommand, log id: 397c9334 >2013-03-07 20:05:00,273 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [1951c47] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:05:00,274 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [1951c47] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 20:05:00,275 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [1951c47] Autorecovering 1 storage domains >2013-03-07 20:05:00,275 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [1951c47] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 20:05:00,276 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [233f1f39] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:05:00,276 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [233f1f39] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 8:05 PM >2013-03-07 20:05:00,279 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [233f1f39] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 8:05 PM >2013-03-07 20:05:02,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [5bdf326d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 587165ba >2013-03-07 20:05:02,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [5bdf326d] FINISH, GetHardwareInfoVDSCommand, log id: 587165ba >2013-03-07 20:05:02,488 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-12) [3b7444b6] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 20:05:02,491 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-12) [2523910d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 20:05:02,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5f3b0dbe >2013-03-07 20:06:07,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 20:06:07,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:06:07,486 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 20:06:07,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 5f3b0dbe >2013-03-07 20:06:07,490 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [27184eb3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:06:07,505 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [27184eb3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c74d541 >2013-03-07 20:06:07,506 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [27184eb3] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 20:06:07,506 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [27184eb3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c74d541 >2013-03-07 20:06:07,510 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [2523910d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 20:06:07,521 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-12) [7b1a7191] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:06:07,522 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [7b1a7191] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6de37156 >2013-03-07 20:06:07,523 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [7b1a7191] FINISH, SetVdsStatusVDSCommand, log id: 6de37156 >2013-03-07 20:06:07,529 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-12) [7b1a7191] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 20:06:07,549 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-12) [1721a4c1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:06:07,552 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-12) [2287daa3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:06:07,553 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-12) [2287daa3] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 20:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [5be7886b] Autorecovering 1 hosts >2013-03-07 20:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [5be7886b] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-51) [62bd7478] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-51) [62bd7478] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [62bd7478] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4740df7d >2013-03-07 20:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [62bd7478] FINISH, SetVdsStatusVDSCommand, log id: 4740df7d >2013-03-07 20:10:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-51) [62bd7478] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6d90852 >2013-03-07 20:10:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [62bd7478] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 55d342c6 >2013-03-07 20:10:00,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [62bd7478] FINISH, GetHardwareInfoVDSCommand, log id: 55d342c6 >2013-03-07 20:10:00,278 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-51) [18ddb966] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:10:00,281 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-51) [3e2d422d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:10:00,282 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-51) [3e2d422d] FINISH, ActivateVdsVDSCommand, log id: 6d90852 >2013-03-07 20:10:00,302 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-51) [3e2d422d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:10:00,303 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-51) [3e2d422d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 20:10:00,304 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [3e2d422d] Autorecovering 1 storage domains >2013-03-07 20:10:00,304 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [3e2d422d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 20:10:00,305 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-51) [6289fa91] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:10:00,305 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-51) [6289fa91] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 8:10 PM >2013-03-07 20:10:00,308 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-51) [6289fa91] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 8:10 PM >2013-03-07 20:10:01,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [113ae533] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4791960e >2013-03-07 20:10:01,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [113ae533] FINISH, GetHardwareInfoVDSCommand, log id: 4791960e >2013-03-07 20:10:01,798 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-94) [168cb444] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 20:10:01,805 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-94) [67635660] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 20:10:01,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1337eacb >2013-03-07 20:11:05,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 20:11:05,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:11:05,815 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 20:11:05,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 1337eacb >2013-03-07 20:11:05,819 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [40a5914a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:11:05,836 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [40a5914a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2685ee14 >2013-03-07 20:11:05,836 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [40a5914a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 20:11:05,837 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [40a5914a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2685ee14 >2013-03-07 20:11:05,842 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [40a5914a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 20:11:05,842 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [67635660] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 20:11:05,851 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-94) [6bb7314c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:11:05,853 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [6bb7314c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 635037a4 >2013-03-07 20:11:05,854 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [6bb7314c] FINISH, SetVdsStatusVDSCommand, log id: 635037a4 >2013-03-07 20:11:05,860 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-94) [6bb7314c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 20:11:05,885 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [31845a67] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:11:05,888 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [1ed6de3e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:11:05,889 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-94) [1ed6de3e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 20:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [262b6da6] Autorecovering 1 hosts >2013-03-07 20:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [262b6da6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [1591245b] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [1591245b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [1591245b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 48db4918 >2013-03-07 20:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [1591245b] FINISH, SetVdsStatusVDSCommand, log id: 48db4918 >2013-03-07 20:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-41) [1591245b] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 48550494 >2013-03-07 20:15:00,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [1591245b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4c59dce1 >2013-03-07 20:15:00,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [1591245b] FINISH, GetHardwareInfoVDSCommand, log id: 4c59dce1 >2013-03-07 20:15:00,450 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-41) [73947b59] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:15:00,453 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-41) [432c5af7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:15:00,457 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-41) [432c5af7] FINISH, ActivateVdsVDSCommand, log id: 48550494 >2013-03-07 20:15:00,481 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [432c5af7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:15:00,482 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-41) [432c5af7] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 20:15:00,483 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [432c5af7] Autorecovering 1 storage domains >2013-03-07 20:15:00,483 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [432c5af7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 20:15:00,484 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [363a8986] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:15:00,484 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [363a8986] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 8:15 PM >2013-03-07 20:15:00,487 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [363a8986] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 8:15 PM >2013-03-07 20:15:00,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [760cbcdf] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 46bd1c54 >2013-03-07 20:15:00,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [760cbcdf] FINISH, GetHardwareInfoVDSCommand, log id: 46bd1c54 >2013-03-07 20:15:00,561 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-72) [1ca98c4c] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 20:15:00,564 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-72) [638dbcbd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 20:15:00,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 10fe7a8c >2013-03-07 20:16:05,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 20:16:05,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:16:05,539 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 20:16:05,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 10fe7a8c >2013-03-07 20:16:05,543 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [7079db8b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:16:05,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [7079db8b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 39b93009 >2013-03-07 20:16:05,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [7079db8b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 20:16:05,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [7079db8b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 39b93009 >2013-03-07 20:16:05,562 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [7079db8b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 20:16:05,562 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [638dbcbd] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 20:16:05,572 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-72) [3c76bc28] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:16:05,574 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [3c76bc28] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 673c4381 >2013-03-07 20:16:05,575 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [3c76bc28] FINISH, SetVdsStatusVDSCommand, log id: 673c4381 >2013-03-07 20:16:05,582 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [3c76bc28] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 20:16:05,601 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [40a5ce7f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:16:05,603 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [484afdc7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:16:05,605 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-72) [484afdc7] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 20:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [28e66ea6] Autorecovering 1 hosts >2013-03-07 20:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [28e66ea6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-48) [2d27abcd] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-48) [2d27abcd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [2d27abcd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1a9e6aa7 >2013-03-07 20:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [2d27abcd] FINISH, SetVdsStatusVDSCommand, log id: 1a9e6aa7 >2013-03-07 20:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-48) [2d27abcd] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 15ae0e5a >2013-03-07 20:20:00,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [2d27abcd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 536bb280 >2013-03-07 20:20:00,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [2d27abcd] FINISH, GetHardwareInfoVDSCommand, log id: 536bb280 >2013-03-07 20:20:00,099 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-48) [322bc64a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:20:00,102 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-48) [13933e17] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:20:00,103 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-48) [13933e17] FINISH, ActivateVdsVDSCommand, log id: 15ae0e5a >2013-03-07 20:20:00,122 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-48) [13933e17] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:20:00,123 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-48) [13933e17] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 20:20:00,124 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [13933e17] Autorecovering 1 storage domains >2013-03-07 20:20:00,125 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [13933e17] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 20:20:00,125 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-48) [9f7f384] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:20:00,125 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-48) [9f7f384] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 8:20 PM >2013-03-07 20:20:00,128 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-48) [9f7f384] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 8:20 PM >2013-03-07 20:20:02,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [68082913] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 72703f79 >2013-03-07 20:20:02,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [68082913] FINISH, GetHardwareInfoVDSCommand, log id: 72703f79 >2013-03-07 20:20:02,858 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-49) [4889a6d2] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 20:20:02,861 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-49) [2167ee3f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 20:20:02,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3cd3f34c >2013-03-07 20:21:07,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 20:21:07,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:21:07,899 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 20:21:07,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 3cd3f34c >2013-03-07 20:21:07,903 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [189b59a9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:21:07,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [189b59a9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6fb60c4a >2013-03-07 20:21:07,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [189b59a9] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 20:21:07,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [189b59a9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6fb60c4a >2013-03-07 20:21:07,921 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [189b59a9] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 20:21:07,921 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [2167ee3f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 20:21:07,931 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-49) [88fb2c3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:21:07,933 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [88fb2c3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7b443a4a >2013-03-07 20:21:07,934 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [88fb2c3] FINISH, SetVdsStatusVDSCommand, log id: 7b443a4a >2013-03-07 20:21:07,940 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-49) [88fb2c3] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 20:21:07,960 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-49) [926042b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:21:07,962 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-49) [50b7f5f1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:21:07,964 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) [50b7f5f1] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 20:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [24e13c2f] Autorecovering 1 hosts >2013-03-07 20:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [24e13c2f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-100) [68fc0bf4] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-100) [68fc0bf4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [68fc0bf4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 58a44810 >2013-03-07 20:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [68fc0bf4] FINISH, SetVdsStatusVDSCommand, log id: 58a44810 >2013-03-07 20:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-100) [68fc0bf4] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 74cf878b >2013-03-07 20:25:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [68fc0bf4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: bd46cf >2013-03-07 20:25:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [68fc0bf4] FINISH, GetHardwareInfoVDSCommand, log id: bd46cf >2013-03-07 20:25:00,195 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-100) [4ac15872] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:25:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-100) [168313da] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:25:00,199 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-100) [168313da] FINISH, ActivateVdsVDSCommand, log id: 74cf878b >2013-03-07 20:25:00,219 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-100) [168313da] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:25:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-100) [168313da] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 20:25:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [168313da] Autorecovering 1 storage domains >2013-03-07 20:25:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [168313da] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 20:25:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-100) [68d98cc8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:25:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-100) [68d98cc8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 8:25 PM >2013-03-07 20:25:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-100) [68d98cc8] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 8:25 PM >2013-03-07 20:25:02,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [1e658023] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4c7e2fd >2013-03-07 20:25:02,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [1e658023] FINISH, GetHardwareInfoVDSCommand, log id: 4c7e2fd >2013-03-07 20:25:02,200 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-97) [3f3f6ae1] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 20:25:02,203 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-97) [73b0bddf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 20:25:02,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 33b5c10 >2013-03-07 20:26:06,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 20:26:06,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:26:06,526 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 20:26:06,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 33b5c10 >2013-03-07 20:26:06,530 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [6e358e11] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:26:06,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [6e358e11] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3457e6 >2013-03-07 20:26:06,545 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [6e358e11] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 20:26:06,545 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [6e358e11] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3457e6 >2013-03-07 20:26:06,547 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [6e358e11] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 20:26:06,547 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [73b0bddf] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 20:26:06,557 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-97) [62b084b7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:26:06,559 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [62b084b7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 19522bd5 >2013-03-07 20:26:06,560 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [62b084b7] FINISH, SetVdsStatusVDSCommand, log id: 19522bd5 >2013-03-07 20:26:06,567 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-97) [62b084b7] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 20:26:06,586 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-97) [75a7c6f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:26:06,589 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-97) [309c8303] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:26:06,590 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-97) [309c8303] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 20:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [eaad4a6] Autorecovering 1 hosts >2013-03-07 20:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [eaad4a6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [6efdc634] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [6efdc634] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [6efdc634] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 640fc2bd >2013-03-07 20:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [6efdc634] FINISH, SetVdsStatusVDSCommand, log id: 640fc2bd >2013-03-07 20:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [6efdc634] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 408f49ce >2013-03-07 20:30:00,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [6efdc634] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5a703b13 >2013-03-07 20:30:00,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [6efdc634] FINISH, GetHardwareInfoVDSCommand, log id: 5a703b13 >2013-03-07 20:30:00,234 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [41ecb79b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:30:00,236 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [5f23d2ae] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:30:00,238 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [5f23d2ae] FINISH, ActivateVdsVDSCommand, log id: 408f49ce >2013-03-07 20:30:00,257 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [5f23d2ae] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:30:00,258 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [5f23d2ae] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 20:30:00,259 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [5f23d2ae] Autorecovering 1 storage domains >2013-03-07 20:30:00,259 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [5f23d2ae] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 20:30:00,260 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [55a7a74b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:30:00,260 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [55a7a74b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 8:30 PM >2013-03-07 20:30:00,263 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [55a7a74b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 8:30 PM >2013-03-07 20:30:00,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [315006da] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2aac11f5 >2013-03-07 20:30:00,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [315006da] FINISH, GetHardwareInfoVDSCommand, log id: 2aac11f5 >2013-03-07 20:30:00,864 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-44) [72136a4a] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 20:30:00,867 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-44) [70a9d3be] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 20:30:00,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 15bd7b31 >2013-03-07 20:30:33,703 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-2) [72589424] Start refreshing all users data >2013-03-07 20:30:34,588 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [2287daa3] Attempting to update VMs/Templates Ovf. >2013-03-07 20:30:34,590 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [2287daa3] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 20:30:34,596 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [2287daa3] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 20:30:34,596 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [2287daa3] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 20:30:34,599 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [2287daa3] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 20:30:34,599 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [2287daa3] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 20:30:34,601 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [2287daa3] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 20:31:05,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 20:31:05,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:31:05,844 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 20:31:05,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 15bd7b31 >2013-03-07 20:31:05,848 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [ad2f16f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:31:05,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [ad2f16f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a42594f >2013-03-07 20:31:05,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [ad2f16f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 20:31:05,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [ad2f16f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a42594f >2013-03-07 20:31:05,866 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [ad2f16f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 20:31:05,866 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [70a9d3be] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 20:31:05,877 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-44) [5048eb59] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:31:05,879 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [5048eb59] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 632e9b61 >2013-03-07 20:31:05,880 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [5048eb59] FINISH, SetVdsStatusVDSCommand, log id: 632e9b61 >2013-03-07 20:31:05,888 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-44) [5048eb59] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 20:31:05,908 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-44) [7b544fb3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:31:05,910 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-44) [3e2c6ba7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:31:05,911 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-44) [3e2c6ba7] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 20:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [2311d9c2] Autorecovering 1 hosts >2013-03-07 20:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [2311d9c2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [412bdda5] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [412bdda5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [412bdda5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3bd89747 >2013-03-07 20:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [412bdda5] FINISH, SetVdsStatusVDSCommand, log id: 3bd89747 >2013-03-07 20:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [412bdda5] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7a20879a >2013-03-07 20:35:00,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [412bdda5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 205f7f72 >2013-03-07 20:35:00,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [412bdda5] FINISH, GetHardwareInfoVDSCommand, log id: 205f7f72 >2013-03-07 20:35:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-96) [2244b347] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:35:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-96) [7a93e05b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:35:00,207 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [7a93e05b] FINISH, ActivateVdsVDSCommand, log id: 7a20879a >2013-03-07 20:35:00,235 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [7a93e05b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:35:00,236 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-96) [7a93e05b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 20:35:00,238 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [7a93e05b] Autorecovering 1 storage domains >2013-03-07 20:35:00,238 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [7a93e05b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 20:35:00,238 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [65dc55aa] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:35:00,238 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [65dc55aa] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 8:35 PM >2013-03-07 20:35:00,241 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [65dc55aa] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 8:35 PM >2013-03-07 20:35:00,303 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [5ea344e8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 22ded967 >2013-03-07 20:35:00,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [5ea344e8] FINISH, GetHardwareInfoVDSCommand, log id: 22ded967 >2013-03-07 20:35:00,414 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-36) [49244406] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 20:35:00,422 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-36) [110aaf64] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 20:35:00,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 745583ed >2013-03-07 20:36:05,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 20:36:05,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:36:05,762 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 20:36:05,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 745583ed >2013-03-07 20:36:05,766 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [53a7203c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:36:05,783 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [53a7203c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 465e6d4d >2013-03-07 20:36:05,783 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [53a7203c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 20:36:05,783 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [53a7203c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 465e6d4d >2013-03-07 20:36:05,786 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [53a7203c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 20:36:05,786 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [110aaf64] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 20:36:05,799 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-36) [209b1a30] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:36:05,801 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [209b1a30] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 25ded754 >2013-03-07 20:36:05,802 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [209b1a30] FINISH, SetVdsStatusVDSCommand, log id: 25ded754 >2013-03-07 20:36:05,811 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [209b1a30] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 20:36:05,833 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [f9652ef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:36:05,836 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [6d06c4f5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:36:05,837 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-36) [6d06c4f5] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 20:40:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [27ea8a22] Autorecovering 1 hosts >2013-03-07 20:40:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [27ea8a22] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:40:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-38) [70ba271a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:40:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-38) [70ba271a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [70ba271a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1c2d315c >2013-03-07 20:40:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [70ba271a] FINISH, SetVdsStatusVDSCommand, log id: 1c2d315c >2013-03-07 20:40:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-38) [70ba271a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2fbc48e6 >2013-03-07 20:40:00,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [70ba271a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 37e52d70 >2013-03-07 20:40:00,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [70ba271a] FINISH, GetHardwareInfoVDSCommand, log id: 37e52d70 >2013-03-07 20:40:00,536 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-38) [bd706da] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:40:00,539 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-38) [bf3458a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:40:00,540 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-38) [bf3458a] FINISH, ActivateVdsVDSCommand, log id: 2fbc48e6 >2013-03-07 20:40:00,563 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-38) [bf3458a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:40:00,564 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-38) [bf3458a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 20:40:00,567 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [bf3458a] Autorecovering 1 storage domains >2013-03-07 20:40:00,567 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [bf3458a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 20:40:00,567 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-38) [11e45fdf] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:40:00,568 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-38) [11e45fdf] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 8:40 PM >2013-03-07 20:40:00,573 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-38) [11e45fdf] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 8:40 PM >2013-03-07 20:40:03,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2a8e635d >2013-03-07 20:40:03,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) FINISH, GetHardwareInfoVDSCommand, log id: 2a8e635d >2013-03-07 20:40:03,084 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-45) [5e5d6d96] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 20:40:03,088 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-45) [43dd75b6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 20:40:03,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 16b0aba1 >2013-03-07 20:41:09,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 20:41:09,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:41:09,084 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 20:41:09,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 16b0aba1 >2013-03-07 20:41:09,088 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [3dd46f36] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:41:09,103 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [3dd46f36] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1a7455e1 >2013-03-07 20:41:09,103 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [3dd46f36] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 20:41:09,103 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [3dd46f36] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1a7455e1 >2013-03-07 20:41:09,105 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [3dd46f36] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 20:41:09,106 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [43dd75b6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 20:41:09,116 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-45) [ae77719] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:41:09,118 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [ae77719] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7bc665ed >2013-03-07 20:41:09,119 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [ae77719] FINISH, SetVdsStatusVDSCommand, log id: 7bc665ed >2013-03-07 20:41:09,124 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-45) [ae77719] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 20:41:09,145 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-45) [ab88da4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:41:09,147 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-45) [7d9f4ed3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:41:09,148 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-45) [7d9f4ed3] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 20:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Autorecovering 1 hosts >2013-03-07 20:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [7b45e91b] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [7b45e91b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [7b45e91b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4315afea >2013-03-07 20:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [7b45e91b] FINISH, SetVdsStatusVDSCommand, log id: 4315afea >2013-03-07 20:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-90) [7b45e91b] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3b9f7af >2013-03-07 20:45:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [7b45e91b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6e1b0049 >2013-03-07 20:45:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [7b45e91b] FINISH, GetHardwareInfoVDSCommand, log id: 6e1b0049 >2013-03-07 20:45:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-90) [5288c0ad] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:45:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-90) [293827f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:45:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-90) [293827f] FINISH, ActivateVdsVDSCommand, log id: 3b9f7af >2013-03-07 20:45:00,111 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [293827f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:45:00,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [293827f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 20:45:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [293827f] Autorecovering 1 storage domains >2013-03-07 20:45:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [293827f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 20:45:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [319f8114] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:45:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [319f8114] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 8:45 PM >2013-03-07 20:45:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [319f8114] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 8:45 PM >2013-03-07 20:45:00,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [ffe013d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3f861df2 >2013-03-07 20:45:00,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [ffe013d] FINISH, GetHardwareInfoVDSCommand, log id: 3f861df2 >2013-03-07 20:45:00,393 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-20) [367d14c0] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 20:45:00,396 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-20) [6570daf2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 20:45:00,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 665d5d6c >2013-03-07 20:46:06,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 20:46:06,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:46:06,429 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 20:46:06,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 665d5d6c >2013-03-07 20:46:06,432 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [612d2bc7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:46:06,447 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [612d2bc7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 10f07f5b >2013-03-07 20:46:06,447 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [612d2bc7] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 20:46:06,447 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [612d2bc7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 10f07f5b >2013-03-07 20:46:06,449 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [612d2bc7] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 20:46:06,450 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [6570daf2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 20:46:06,460 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-20) [33e7ea63] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:46:06,462 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [33e7ea63] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3471ce39 >2013-03-07 20:46:06,463 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [33e7ea63] FINISH, SetVdsStatusVDSCommand, log id: 3471ce39 >2013-03-07 20:46:06,469 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-20) [33e7ea63] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 20:46:06,489 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-20) [74c0c0fb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:46:06,491 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-20) [42c286c1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:46:06,493 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-20) [42c286c1] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 20:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) Autorecovering 1 hosts >2013-03-07 20:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-30) [46601662] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-30) [46601662] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-30) [46601662] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7c1402fd >2013-03-07 20:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-30) [46601662] FINISH, SetVdsStatusVDSCommand, log id: 7c1402fd >2013-03-07 20:50:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-30) [46601662] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1383965e >2013-03-07 20:50:00,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-30) [46601662] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2a959b83 >2013-03-07 20:50:00,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-30) [46601662] FINISH, GetHardwareInfoVDSCommand, log id: 2a959b83 >2013-03-07 20:50:00,095 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-30) [1209269e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:50:00,098 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-30) [47e95c7a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:50:00,100 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-30) [47e95c7a] FINISH, ActivateVdsVDSCommand, log id: 1383965e >2013-03-07 20:50:00,119 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-30) [47e95c7a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:50:00,120 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-30) [47e95c7a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 20:50:00,122 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) [47e95c7a] Autorecovering 1 storage domains >2013-03-07 20:50:00,122 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) [47e95c7a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 20:50:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-30) [3fbbb1fc] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:50:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-30) [3fbbb1fc] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 8:50 PM >2013-03-07 20:50:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-30) [3fbbb1fc] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 8:50 PM >2013-03-07 20:50:00,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [3f8dc449] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: a11dc0f >2013-03-07 20:50:00,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [3f8dc449] FINISH, GetHardwareInfoVDSCommand, log id: a11dc0f >2013-03-07 20:50:00,731 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-78) [6cc94009] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 20:50:00,734 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-78) [30cf0b83] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 20:50:00,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 47c7f56c >2013-03-07 20:51:05,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 20:51:05,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:51:05,770 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 20:51:05,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 47c7f56c >2013-03-07 20:51:05,774 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [6ef41942] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:51:05,789 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [6ef41942] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 57cea0ae >2013-03-07 20:51:05,789 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [6ef41942] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 20:51:05,789 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [6ef41942] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 57cea0ae >2013-03-07 20:51:05,791 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [6ef41942] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 20:51:05,791 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [30cf0b83] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 20:51:05,803 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-78) [22ace136] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:51:05,804 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [22ace136] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5c1320ab >2013-03-07 20:51:05,806 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [22ace136] FINISH, SetVdsStatusVDSCommand, log id: 5c1320ab >2013-03-07 20:51:05,812 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-78) [22ace136] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 20:51:05,834 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-78) [6b05f6ed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:51:05,837 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-78) [722e1bc0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:51:05,838 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-78) [722e1bc0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 20:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [9cd5977] Autorecovering 1 hosts >2013-03-07 20:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [9cd5977] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [e15b825] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [e15b825] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [e15b825] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 62f4dea1 >2013-03-07 20:55:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [e15b825] FINISH, SetVdsStatusVDSCommand, log id: 62f4dea1 >2013-03-07 20:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [e15b825] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2c162918 >2013-03-07 20:55:00,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [e15b825] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2a66c986 >2013-03-07 20:55:00,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [e15b825] FINISH, GetHardwareInfoVDSCommand, log id: 2a66c986 >2013-03-07 20:55:00,212 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [4cce0216] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:55:00,215 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [25e2464c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:55:00,216 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [25e2464c] FINISH, ActivateVdsVDSCommand, log id: 2c162918 >2013-03-07 20:55:00,239 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [25e2464c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 20:55:00,240 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [25e2464c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 20:55:00,241 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [25e2464c] Autorecovering 1 storage domains >2013-03-07 20:55:00,241 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [25e2464c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 20:55:00,242 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [11dfe85f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:55:00,242 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [11dfe85f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 8:55 PM >2013-03-07 20:55:00,245 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [11dfe85f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 8:55 PM >2013-03-07 20:55:03,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [c6fb43f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7dfffa50 >2013-03-07 20:55:03,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [c6fb43f] FINISH, GetHardwareInfoVDSCommand, log id: 7dfffa50 >2013-03-07 20:55:03,083 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-79) [7994975] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 20:55:03,087 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-79) [4fb3a3c3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 20:55:03,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1abb7cc5 >2013-03-07 20:56:09,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 20:56:09,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 20:56:09,108 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 20:56:09,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 1abb7cc5 >2013-03-07 20:56:09,112 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [1ecffb9c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 20:56:09,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1ecffb9c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 59830b19 >2013-03-07 20:56:09,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [1ecffb9c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 20:56:09,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1ecffb9c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 59830b19 >2013-03-07 20:56:09,129 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [1ecffb9c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 20:56:09,129 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [4fb3a3c3] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 20:56:09,140 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-79) [18298d46] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:56:09,169 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [18298d46] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7a10a60a >2013-03-07 20:56:09,171 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [18298d46] FINISH, SetVdsStatusVDSCommand, log id: 7a10a60a >2013-03-07 20:56:09,178 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-79) [18298d46] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 20:56:09,198 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-79) [1d1fb428] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:56:09,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-79) [2ef4d8bf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 20:56:09,202 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-79) [2ef4d8bf] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 21:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) Autorecovering 1 hosts >2013-03-07 21:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [79a0881b] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [79a0881b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-29) [79a0881b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5eaf3e26 >2013-03-07 21:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-29) [79a0881b] FINISH, SetVdsStatusVDSCommand, log id: 5eaf3e26 >2013-03-07 21:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-29) [79a0881b] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5189ee9f >2013-03-07 21:00:00,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-29) [79a0881b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1655332f >2013-03-07 21:00:00,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-29) [79a0881b] FINISH, GetHardwareInfoVDSCommand, log id: 1655332f >2013-03-07 21:00:00,215 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-29) [3ad219a9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:00:00,218 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-29) [5a45b431] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:00:00,219 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-29) [5a45b431] FINISH, ActivateVdsVDSCommand, log id: 5189ee9f >2013-03-07 21:00:00,239 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [5a45b431] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:00:00,240 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-29) [5a45b431] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 21:00:00,241 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [5a45b431] Autorecovering 1 storage domains >2013-03-07 21:00:00,241 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [5a45b431] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 21:00:00,242 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [7d798295] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:00:00,242 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [7d798295] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 9:00 PM >2013-03-07 21:00:00,245 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [7d798295] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 9:00 PM >2013-03-07 21:00:00,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [61ae1732] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 32d8ee17 >2013-03-07 21:00:00,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [61ae1732] FINISH, GetHardwareInfoVDSCommand, log id: 32d8ee17 >2013-03-07 21:00:00,451 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-4) [66cb521c] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 21:00:00,454 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-4) [4b3f6222] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 21:00:00,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 12ec7979 >2013-03-07 21:01:05,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 21:01:05,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:01:05,457 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 21:01:05,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 12ec7979 >2013-03-07 21:01:05,460 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [3ca8aeaf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:01:05,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [3ca8aeaf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 54f448f5 >2013-03-07 21:01:05,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [3ca8aeaf] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 21:01:05,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [3ca8aeaf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 54f448f5 >2013-03-07 21:01:05,477 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [3ca8aeaf] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 21:01:05,477 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [4b3f6222] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 21:01:05,497 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-4) [439df110] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:01:05,499 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [439df110] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3846a769 >2013-03-07 21:01:05,500 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [439df110] FINISH, SetVdsStatusVDSCommand, log id: 3846a769 >2013-03-07 21:01:05,506 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-4) [439df110] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 21:01:05,525 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-4) [49e4a44] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:01:05,528 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-4) [10c1773d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:01:05,529 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-4) [10c1773d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 21:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [75dd7f53] Autorecovering 1 hosts >2013-03-07 21:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [75dd7f53] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [2925aa3f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [2925aa3f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [2925aa3f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 72335f69 >2013-03-07 21:05:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [2925aa3f] FINISH, SetVdsStatusVDSCommand, log id: 72335f69 >2013-03-07 21:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [2925aa3f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 545200a8 >2013-03-07 21:05:00,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [2925aa3f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5fcbc179 >2013-03-07 21:05:00,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [2925aa3f] FINISH, GetHardwareInfoVDSCommand, log id: 5fcbc179 >2013-03-07 21:05:00,257 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [79c3e89a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:05:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [2e205ccf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:05:00,261 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [2e205ccf] FINISH, ActivateVdsVDSCommand, log id: 545200a8 >2013-03-07 21:05:00,281 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [2e205ccf] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:05:00,282 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-52) [2e205ccf] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 21:05:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [2e205ccf] Autorecovering 1 storage domains >2013-03-07 21:05:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [2e205ccf] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 21:05:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [7754de2e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:05:00,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [7754de2e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 9:05 PM >2013-03-07 21:05:00,292 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [7754de2e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 9:05 PM >2013-03-07 21:05:02,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [4d9a0ab0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 799e181a >2013-03-07 21:05:02,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [4d9a0ab0] FINISH, GetHardwareInfoVDSCommand, log id: 799e181a >2013-03-07 21:05:02,776 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-19) [64c4fa3a] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 21:05:02,779 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-19) [a923cf9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 21:05:02,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4ec4b6e6 >2013-03-07 21:06:07,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 21:06:07,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:06:07,772 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 21:06:07,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 4ec4b6e6 >2013-03-07 21:06:07,776 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [7435cdc2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:06:07,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [7435cdc2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5cd42a >2013-03-07 21:06:07,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [7435cdc2] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 21:06:07,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [7435cdc2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5cd42a >2013-03-07 21:06:07,796 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [a923cf9] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 21:06:07,805 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-19) [1647cc3b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:06:07,807 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [1647cc3b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 59a1e786 >2013-03-07 21:06:07,808 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [1647cc3b] FINISH, SetVdsStatusVDSCommand, log id: 59a1e786 >2013-03-07 21:06:07,815 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-19) [1647cc3b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 21:06:07,834 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-19) [5083c570] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:06:07,837 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-19) [394ce138] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:06:07,838 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-19) [394ce138] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 21:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) Autorecovering 1 hosts >2013-03-07 21:10:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [78f54906] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [78f54906] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [78f54906] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 344e055 >2013-03-07 21:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [78f54906] FINISH, SetVdsStatusVDSCommand, log id: 344e055 >2013-03-07 21:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-22) [78f54906] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1918c5b8 >2013-03-07 21:10:00,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [78f54906] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7dc39639 >2013-03-07 21:10:00,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [78f54906] FINISH, GetHardwareInfoVDSCommand, log id: 7dc39639 >2013-03-07 21:10:00,258 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [6c4ab279] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:10:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [64513660] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:10:00,262 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-22) [64513660] FINISH, ActivateVdsVDSCommand, log id: 1918c5b8 >2013-03-07 21:10:00,282 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [64513660] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:10:00,283 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [64513660] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 21:10:00,291 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [64513660] Autorecovering 1 storage domains >2013-03-07 21:10:00,291 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [64513660] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 21:10:00,292 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [493d2a36] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:10:00,292 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [493d2a36] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 9:10 PM >2013-03-07 21:10:00,295 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [493d2a36] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 9:10 PM >2013-03-07 21:10:02,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [4ffad86] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7154d790 >2013-03-07 21:10:02,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [4ffad86] FINISH, GetHardwareInfoVDSCommand, log id: 7154d790 >2013-03-07 21:10:02,111 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-27) [1e76ac49] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 21:10:02,114 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-27) [74669591] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 21:10:02,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: d6bb81e >2013-03-07 21:11:08,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 21:11:08,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:11:08,115 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 21:11:08,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: d6bb81e >2013-03-07 21:11:08,119 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [33223cc8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:11:08,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [33223cc8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 228982a8 >2013-03-07 21:11:08,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [33223cc8] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 21:11:08,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [33223cc8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 228982a8 >2013-03-07 21:11:08,137 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [33223cc8] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 21:11:08,137 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [74669591] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 21:11:08,148 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-27) [4a08ae21] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:11:08,150 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [4a08ae21] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 395a66ba >2013-03-07 21:11:08,151 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [4a08ae21] FINISH, SetVdsStatusVDSCommand, log id: 395a66ba >2013-03-07 21:11:08,158 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-27) [4a08ae21] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 21:11:08,178 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-27) [10619ea4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:11:08,180 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-27) [22997f1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:11:08,182 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-27) [22997f1] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 21:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [253426d6] Autorecovering 1 hosts >2013-03-07 21:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [253426d6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-5) [66f18803] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-5) [66f18803] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [66f18803] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 42c0a0ed >2013-03-07 21:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [66f18803] FINISH, SetVdsStatusVDSCommand, log id: 42c0a0ed >2013-03-07 21:15:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-5) [66f18803] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6e24664c >2013-03-07 21:15:00,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [66f18803] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1d79e4b4 >2013-03-07 21:15:00,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [66f18803] FINISH, GetHardwareInfoVDSCommand, log id: 1d79e4b4 >2013-03-07 21:15:00,301 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-5) [771f01ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:15:00,308 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-5) [455acbca] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:15:00,310 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-5) [455acbca] FINISH, ActivateVdsVDSCommand, log id: 6e24664c >2013-03-07 21:15:00,330 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-5) [455acbca] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:15:00,334 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [455acbca] Autorecovering 1 storage domains >2013-03-07 21:15:00,334 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [455acbca] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 21:15:00,335 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-5) [22438700] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:15:00,335 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-5) [22438700] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 9:15 PM >2013-03-07 21:15:00,338 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-5) [22438700] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 9:15 PM >2013-03-07 21:15:02,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [27036916] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 55b7d0f3 >2013-03-07 21:15:02,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [27036916] FINISH, GetHardwareInfoVDSCommand, log id: 55b7d0f3 >2013-03-07 21:15:02,422 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-9) [37c81669] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 21:15:02,426 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-9) [53fcb7c2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 21:15:02,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 43e15c1 >2013-03-07 21:16:07,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 21:16:07,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:16:07,423 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 21:16:07,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 43e15c1 >2013-03-07 21:16:07,427 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [56c649d7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:16:07,441 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [56c649d7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4e8c7735 >2013-03-07 21:16:07,441 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [56c649d7] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 21:16:07,441 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [56c649d7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4e8c7735 >2013-03-07 21:16:07,444 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [56c649d7] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 21:16:07,444 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [53fcb7c2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 21:16:07,454 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-9) [749de7fb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:16:07,455 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [749de7fb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 39fd03dc >2013-03-07 21:16:07,457 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [749de7fb] FINISH, SetVdsStatusVDSCommand, log id: 39fd03dc >2013-03-07 21:16:07,463 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-9) [749de7fb] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 21:16:07,483 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-9) [33dc6e99] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:16:07,485 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-9) [74e312c1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:16:07,487 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-9) [74e312c1] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 21:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [233f1f39] Autorecovering 1 hosts >2013-03-07 21:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [233f1f39] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [2381525d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [2381525d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [2381525d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: c321acb >2013-03-07 21:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [2381525d] FINISH, SetVdsStatusVDSCommand, log id: c321acb >2013-03-07 21:20:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [2381525d] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 41abd2d6 >2013-03-07 21:20:00,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [2381525d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1a2d3919 >2013-03-07 21:20:00,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [2381525d] FINISH, GetHardwareInfoVDSCommand, log id: 1a2d3919 >2013-03-07 21:20:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [556aaa65] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:20:00,206 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [2a50bbec] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:20:00,208 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [2a50bbec] FINISH, ActivateVdsVDSCommand, log id: 41abd2d6 >2013-03-07 21:20:00,227 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [2a50bbec] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:20:00,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [2a50bbec] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 21:20:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [2a50bbec] Autorecovering 1 storage domains >2013-03-07 21:20:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [2a50bbec] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 21:20:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [3a994fb] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:20:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [3a994fb] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 9:20 PM >2013-03-07 21:20:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [3a994fb] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 9:20 PM >2013-03-07 21:20:01,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [4b2ef732] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 787030b >2013-03-07 21:20:01,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [4b2ef732] FINISH, GetHardwareInfoVDSCommand, log id: 787030b >2013-03-07 21:20:01,735 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-47) [53ba2ac8] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 21:20:01,738 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-47) [1552418a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 21:20:01,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6764022a >2013-03-07 21:21:05,768 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 21:21:05,768 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:21:05,768 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 21:21:05,768 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 6764022a >2013-03-07 21:21:05,772 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [205c574d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:21:05,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [205c574d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4823830a >2013-03-07 21:21:05,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [205c574d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 21:21:05,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [205c574d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4823830a >2013-03-07 21:21:05,793 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [205c574d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 21:21:05,793 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [1552418a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 21:21:05,804 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-47) [5638bc64] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:21:05,805 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [5638bc64] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4275ce12 >2013-03-07 21:21:05,806 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [5638bc64] FINISH, SetVdsStatusVDSCommand, log id: 4275ce12 >2013-03-07 21:21:05,813 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [5638bc64] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 21:21:05,832 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [9d0b4da] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:21:05,835 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [7581f99e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:21:05,836 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-47) [7581f99e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 21:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [553221d9] Autorecovering 1 hosts >2013-03-07 21:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [553221d9] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [59da3437] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [59da3437] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [59da3437] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 715daf77 >2013-03-07 21:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [59da3437] FINISH, SetVdsStatusVDSCommand, log id: 715daf77 >2013-03-07 21:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-70) [59da3437] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 750605d9 >2013-03-07 21:25:00,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [59da3437] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 246830ad >2013-03-07 21:25:00,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [59da3437] FINISH, GetHardwareInfoVDSCommand, log id: 246830ad >2013-03-07 21:25:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-70) [5c5d370c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:25:00,206 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-70) [e36b852] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:25:00,207 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-70) [e36b852] FINISH, ActivateVdsVDSCommand, log id: 750605d9 >2013-03-07 21:25:00,227 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [e36b852] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:25:00,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-70) [e36b852] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 21:25:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [e36b852] Autorecovering 1 storage domains >2013-03-07 21:25:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [e36b852] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 21:25:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [6b31e9fe] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:25:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [6b31e9fe] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 9:25 PM >2013-03-07 21:25:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [6b31e9fe] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 9:25 PM >2013-03-07 21:25:03,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [4838c2ca] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5f5a16cc >2013-03-07 21:25:03,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [4838c2ca] FINISH, GetHardwareInfoVDSCommand, log id: 5f5a16cc >2013-03-07 21:25:03,084 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-61) [3a1f7714] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 21:25:03,087 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-61) [28306c7b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 21:25:03,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7fb6574a >2013-03-07 21:26:08,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 21:26:08,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:26:08,096 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 21:26:08,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 7fb6574a >2013-03-07 21:26:08,100 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [442cb502] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:26:08,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [442cb502] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: f04bed5 >2013-03-07 21:26:08,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [442cb502] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 21:26:08,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [442cb502] FINISH, MarkPoolInReconstructModeVDSCommand, log id: f04bed5 >2013-03-07 21:26:08,120 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [442cb502] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 21:26:08,120 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [28306c7b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 21:26:08,130 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-61) [46d6400c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:26:08,131 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [46d6400c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 35561189 >2013-03-07 21:26:08,133 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [46d6400c] FINISH, SetVdsStatusVDSCommand, log id: 35561189 >2013-03-07 21:26:08,139 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-61) [46d6400c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 21:26:08,159 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-61) [76a3dd48] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:26:08,162 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-61) [657cd278] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:26:08,163 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-61) [657cd278] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 21:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [28ad0e2a] Autorecovering 1 hosts >2013-03-07 21:30:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [28ad0e2a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [1892f809] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [1892f809] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [1892f809] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1ee21794 >2013-03-07 21:30:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [1892f809] FINISH, SetVdsStatusVDSCommand, log id: 1ee21794 >2013-03-07 21:30:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-46) [1892f809] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 51a531d3 >2013-03-07 21:30:00,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [1892f809] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3fa11ec3 >2013-03-07 21:30:00,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [1892f809] FINISH, GetHardwareInfoVDSCommand, log id: 3fa11ec3 >2013-03-07 21:30:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-46) [6e5adff4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:30:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-46) [e2a0719] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:30:00,206 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-46) [e2a0719] FINISH, ActivateVdsVDSCommand, log id: 51a531d3 >2013-03-07 21:30:00,226 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [e2a0719] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:30:00,227 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-46) [e2a0719] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 21:30:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [e2a0719] Autorecovering 1 storage domains >2013-03-07 21:30:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [e2a0719] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 21:30:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [497a2e37] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:30:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [497a2e37] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 9:30 PM >2013-03-07 21:30:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [497a2e37] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 9:30 PM >2013-03-07 21:30:02,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [1760c683] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1f45c078 >2013-03-07 21:30:02,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [1760c683] FINISH, GetHardwareInfoVDSCommand, log id: 1f45c078 >2013-03-07 21:30:02,406 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-81) [2234a14b] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 21:30:02,410 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-81) [62bcdbf2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 21:30:02,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 295a7b67 >2013-03-07 21:30:33,706 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-77) [25cefe9] Start refreshing all users data >2013-03-07 21:30:34,601 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-19) [394ce138] Attempting to update VMs/Templates Ovf. >2013-03-07 21:30:34,603 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-19) [394ce138] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 21:30:34,609 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-19) [394ce138] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 21:30:34,609 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-19) [394ce138] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 21:30:34,612 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-19) [394ce138] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 21:30:34,612 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-19) [394ce138] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 21:30:34,613 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-19) [394ce138] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 21:31:06,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 21:31:06,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:31:06,483 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 21:31:06,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 295a7b67 >2013-03-07 21:31:06,486 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [29d6633f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:31:06,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [29d6633f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2dfed193 >2013-03-07 21:31:06,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [29d6633f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 21:31:06,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [29d6633f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2dfed193 >2013-03-07 21:31:06,505 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [29d6633f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 21:31:06,505 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [62bcdbf2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 21:31:06,517 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-81) [64c6d470] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:31:06,519 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [64c6d470] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 43fbf808 >2013-03-07 21:31:06,520 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [64c6d470] FINISH, SetVdsStatusVDSCommand, log id: 43fbf808 >2013-03-07 21:31:06,526 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-81) [64c6d470] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 21:31:06,546 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-81) [201da0c5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:31:06,548 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-81) [2385e91c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:31:06,550 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-81) [2385e91c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 21:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [6d06c4f5] Autorecovering 1 hosts >2013-03-07 21:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [6d06c4f5] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [2f22e6d7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [2f22e6d7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [2f22e6d7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 52a3b301 >2013-03-07 21:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [2f22e6d7] FINISH, SetVdsStatusVDSCommand, log id: 52a3b301 >2013-03-07 21:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [2f22e6d7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5d6f09d6 >2013-03-07 21:35:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [2f22e6d7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3f218a97 >2013-03-07 21:35:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [2f22e6d7] FINISH, GetHardwareInfoVDSCommand, log id: 3f218a97 >2013-03-07 21:35:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [7aab184e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:35:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [691cb959] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:35:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [691cb959] FINISH, ActivateVdsVDSCommand, log id: 5d6f09d6 >2013-03-07 21:35:00,222 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [691cb959] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:35:00,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [691cb959] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 21:35:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [691cb959] Autorecovering 1 storage domains >2013-03-07 21:35:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [691cb959] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 21:35:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [62cc6576] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:35:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [62cc6576] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 9:35 PM >2013-03-07 21:35:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [62cc6576] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 9:35 PM >2013-03-07 21:35:00,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [6289fa91] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5afe5a7f >2013-03-07 21:35:00,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [6289fa91] FINISH, GetHardwareInfoVDSCommand, log id: 5afe5a7f >2013-03-07 21:35:00,823 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-51) [3669bd15] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 21:35:00,827 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-51) [264af54d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 21:35:00,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1d517726 >2013-03-07 21:36:05,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 21:36:05,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:36:05,820 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 21:36:05,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 1d517726 >2013-03-07 21:36:05,824 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [437e8754] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:36:05,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [437e8754] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4fde8125 >2013-03-07 21:36:05,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [437e8754] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 21:36:05,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [437e8754] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4fde8125 >2013-03-07 21:36:05,841 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [437e8754] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 21:36:05,842 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [264af54d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 21:36:05,852 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-51) [386448b5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:36:05,853 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [386448b5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1bb40665 >2013-03-07 21:36:05,855 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [386448b5] FINISH, SetVdsStatusVDSCommand, log id: 1bb40665 >2013-03-07 21:36:05,861 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-51) [386448b5] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 21:36:05,882 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-51) [6dcefcbe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:36:05,885 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-51) [45054fea] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:36:05,886 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-51) [45054fea] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 21:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [35751517] Autorecovering 1 hosts >2013-03-07 21:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [35751517] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-15) [66927dd3] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-15) [66927dd3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [66927dd3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7513de18 >2013-03-07 21:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [66927dd3] FINISH, SetVdsStatusVDSCommand, log id: 7513de18 >2013-03-07 21:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-15) [66927dd3] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1c75db4a >2013-03-07 21:40:00,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [66927dd3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2444b827 >2013-03-07 21:40:00,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [66927dd3] FINISH, GetHardwareInfoVDSCommand, log id: 2444b827 >2013-03-07 21:40:00,257 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-15) [1d71310a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:40:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-15) [6e883370] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:40:00,261 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-15) [6e883370] FINISH, ActivateVdsVDSCommand, log id: 1c75db4a >2013-03-07 21:40:00,281 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-15) [6e883370] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:40:00,282 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-15) [6e883370] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 21:40:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [6e883370] Autorecovering 1 storage domains >2013-03-07 21:40:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [6e883370] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 21:40:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-15) [66bbed93] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:40:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-15) [66bbed93] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 9:40 PM >2013-03-07 21:40:00,287 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-15) [66bbed93] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 9:40 PM >2013-03-07 21:40:00,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [2beee732] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4381d7ac >2013-03-07 21:40:00,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [2beee732] FINISH, GetHardwareInfoVDSCommand, log id: 4381d7ac >2013-03-07 21:40:00,361 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-99) [56ab04b5] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 21:40:00,365 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-99) [509d2817] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 21:40:00,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2bcf4c0c >2013-03-07 21:41:04,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 21:41:04,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:41:04,372 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 21:41:04,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 2bcf4c0c >2013-03-07 21:41:04,376 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [56a22620] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:41:04,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [56a22620] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6f4da992 >2013-03-07 21:41:04,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [56a22620] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 21:41:04,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [56a22620] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6f4da992 >2013-03-07 21:41:04,393 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [56a22620] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 21:41:04,394 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [509d2817] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 21:41:04,403 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-99) [48d77eb3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:41:04,405 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [48d77eb3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 10bdeaf8 >2013-03-07 21:41:04,406 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [48d77eb3] FINISH, SetVdsStatusVDSCommand, log id: 10bdeaf8 >2013-03-07 21:41:04,412 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-99) [48d77eb3] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 21:41:04,434 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-99) [751991c9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:41:04,436 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-99) [f65bd94] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:41:04,438 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-99) [f65bd94] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 21:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [42c286c1] Autorecovering 1 hosts >2013-03-07 21:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [42c286c1] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [79116742] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [79116742] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [79116742] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 233c6f42 >2013-03-07 21:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [79116742] FINISH, SetVdsStatusVDSCommand, log id: 233c6f42 >2013-03-07 21:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-20) [79116742] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 322e298c >2013-03-07 21:45:00,303 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [79116742] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6666b897 >2013-03-07 21:45:00,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [79116742] FINISH, GetHardwareInfoVDSCommand, log id: 6666b897 >2013-03-07 21:45:00,341 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-20) [762c0bff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:45:00,344 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-20) [5396a094] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:45:00,345 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-20) [5396a094] FINISH, ActivateVdsVDSCommand, log id: 322e298c >2013-03-07 21:45:00,364 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [5396a094] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:45:00,365 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-20) [5396a094] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 21:45:00,366 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [5396a094] Autorecovering 1 storage domains >2013-03-07 21:45:00,366 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [5396a094] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 21:45:00,367 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [d34ddb5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:45:00,367 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [d34ddb5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 9:45 PM >2013-03-07 21:45:00,370 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [d34ddb5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 9:45 PM >2013-03-07 21:45:01,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [55e61d48] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 349af55c >2013-03-07 21:45:01,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [55e61d48] FINISH, GetHardwareInfoVDSCommand, log id: 349af55c >2013-03-07 21:45:01,687 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-93) [51d0fe0b] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 21:45:01,691 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-93) [20fcdb99] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 21:45:01,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 46c71a09 >2013-03-07 21:46:07,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 21:46:07,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:46:07,682 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 21:46:07,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 46c71a09 >2013-03-07 21:46:07,685 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [56239c71] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:46:07,708 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [56239c71] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 64ac55d7 >2013-03-07 21:46:07,708 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [56239c71] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 21:46:07,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [56239c71] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 64ac55d7 >2013-03-07 21:46:07,711 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [56239c71] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 21:46:07,714 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [20fcdb99] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 21:46:07,722 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-93) [29bb9c0b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:46:07,724 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [29bb9c0b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7ffc2cc3 >2013-03-07 21:46:07,725 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [29bb9c0b] FINISH, SetVdsStatusVDSCommand, log id: 7ffc2cc3 >2013-03-07 21:46:07,732 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-93) [29bb9c0b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 21:46:07,751 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [25999c2a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:46:07,753 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [db405a5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:46:07,755 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-93) [db405a5] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 21:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [65dc55aa] Autorecovering 1 hosts >2013-03-07 21:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [65dc55aa] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [6108e90c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [6108e90c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [6108e90c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1cfbbf1d >2013-03-07 21:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [6108e90c] FINISH, SetVdsStatusVDSCommand, log id: 1cfbbf1d >2013-03-07 21:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [6108e90c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 507479c8 >2013-03-07 21:50:00,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [6108e90c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 74b1c73c >2013-03-07 21:50:00,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [6108e90c] FINISH, GetHardwareInfoVDSCommand, log id: 74b1c73c >2013-03-07 21:50:00,094 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-96) [7b3574af] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:50:00,096 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-96) [264b796f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:50:00,097 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [264b796f] FINISH, ActivateVdsVDSCommand, log id: 507479c8 >2013-03-07 21:50:00,117 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [264b796f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:50:00,117 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-96) [264b796f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 21:50:00,119 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [264b796f] Autorecovering 1 storage domains >2013-03-07 21:50:00,119 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [264b796f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 21:50:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [13b6715d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:50:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [13b6715d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 9:50 PM >2013-03-07 21:50:00,122 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [13b6715d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 9:50 PM >2013-03-07 21:50:01,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-98) [19a5f55] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 44eedcf9 >2013-03-07 21:50:01,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-98) [19a5f55] FINISH, GetHardwareInfoVDSCommand, log id: 44eedcf9 >2013-03-07 21:50:01,995 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-98) [6267c54b] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 21:50:01,998 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-98) [3765e8c4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 21:50:02,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6fec66b1 >2013-03-07 21:51:08,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 21:51:08,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:51:08,024 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 21:51:08,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 6fec66b1 >2013-03-07 21:51:08,027 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [50d8ab25] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:51:08,042 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [50d8ab25] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5793f388 >2013-03-07 21:51:08,042 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [50d8ab25] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 21:51:08,042 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [50d8ab25] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5793f388 >2013-03-07 21:51:08,044 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [50d8ab25] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 21:51:08,045 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [3765e8c4] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 21:51:08,054 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-98) [5da69cf7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:51:08,055 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-98) [5da69cf7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2960dd6f >2013-03-07 21:51:08,057 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-98) [5da69cf7] FINISH, SetVdsStatusVDSCommand, log id: 2960dd6f >2013-03-07 21:51:08,063 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-98) [5da69cf7] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 21:51:08,082 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-98) [b741eb7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:51:08,085 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-98) [2369fac0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:51:08,086 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-98) [2369fac0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 21:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [5f57ee5e] Autorecovering 1 hosts >2013-03-07 21:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [5f57ee5e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [495c0fbd] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [495c0fbd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [495c0fbd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 62241d88 >2013-03-07 21:55:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [495c0fbd] FINISH, SetVdsStatusVDSCommand, log id: 62241d88 >2013-03-07 21:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-50) [495c0fbd] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6c40a866 >2013-03-07 21:55:00,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [495c0fbd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1d5c9cd9 >2013-03-07 21:55:00,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [495c0fbd] FINISH, GetHardwareInfoVDSCommand, log id: 1d5c9cd9 >2013-03-07 21:55:00,590 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-50) [2848de13] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:55:00,593 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-50) [3779669c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:55:00,595 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-50) [3779669c] FINISH, ActivateVdsVDSCommand, log id: 6c40a866 >2013-03-07 21:55:00,614 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [3779669c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 21:55:00,615 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-50) [3779669c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 21:55:00,616 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [3779669c] Autorecovering 1 storage domains >2013-03-07 21:55:00,616 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [3779669c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 21:55:00,617 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [5ccf95a5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:55:00,617 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [5ccf95a5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 9:55 PM >2013-03-07 21:55:00,620 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [5ccf95a5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 9:55 PM >2013-03-07 21:55:02,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1baae62a >2013-03-07 21:55:02,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) FINISH, GetHardwareInfoVDSCommand, log id: 1baae62a >2013-03-07 21:55:02,327 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-37) [57038bed] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 21:55:02,331 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-37) [9cf00f3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 21:55:02,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: e2cec39 >2013-03-07 21:56:06,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 21:56:06,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 21:56:06,337 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 21:56:06,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: e2cec39 >2013-03-07 21:56:06,341 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [5bc9b726] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 21:56:06,356 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [5bc9b726] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 13743931 >2013-03-07 21:56:06,356 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [5bc9b726] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 21:56:06,356 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [5bc9b726] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 13743931 >2013-03-07 21:56:06,358 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [5bc9b726] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 21:56:06,359 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [9cf00f3] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 21:56:06,370 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-37) [7106a11] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:56:06,372 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [7106a11] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 68f641a8 >2013-03-07 21:56:06,373 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [7106a11] FINISH, SetVdsStatusVDSCommand, log id: 68f641a8 >2013-03-07 21:56:06,379 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-37) [7106a11] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 21:56:06,407 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-37) [7f8c77df] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:56:06,409 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-37) [2bd92ca5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 21:56:06,411 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-37) [2bd92ca5] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 22:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [3661b1b5] Autorecovering 1 hosts >2013-03-07 22:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [3661b1b5] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-86) [20127eb2] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-86) [20127eb2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:00:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [20127eb2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 108e0a59 >2013-03-07 22:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [20127eb2] FINISH, SetVdsStatusVDSCommand, log id: 108e0a59 >2013-03-07 22:00:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-86) [20127eb2] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 38a7e1bc >2013-03-07 22:00:00,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [20127eb2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 13597043 >2013-03-07 22:00:00,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [20127eb2] FINISH, GetHardwareInfoVDSCommand, log id: 13597043 >2013-03-07 22:00:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-86) [33447d1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:00:00,095 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-86) [55fd52ab] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:00:00,096 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-86) [55fd52ab] FINISH, ActivateVdsVDSCommand, log id: 38a7e1bc >2013-03-07 22:00:00,115 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-86) [55fd52ab] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:00:00,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-86) [55fd52ab] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 22:00:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [55fd52ab] Autorecovering 1 storage domains >2013-03-07 22:00:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [55fd52ab] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 22:00:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-86) [4ec30a2] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:00:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-86) [4ec30a2] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 10:00 PM >2013-03-07 22:00:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-86) [4ec30a2] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 10:00 PM >2013-03-07 22:00:00,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [10c1773d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 184f036d >2013-03-07 22:00:00,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [10c1773d] FINISH, GetHardwareInfoVDSCommand, log id: 184f036d >2013-03-07 22:00:00,647 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-4) [57b076f7] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 22:00:00,651 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-4) [227224a7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 22:00:00,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1b6f4500 >2013-03-07 22:01:05,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 22:01:05,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:01:05,636 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 22:01:05,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 1b6f4500 >2013-03-07 22:01:05,639 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [10d91be] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:01:05,654 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [10d91be] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 36c4d038 >2013-03-07 22:01:05,654 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [10d91be] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 22:01:05,654 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [10d91be] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 36c4d038 >2013-03-07 22:01:05,656 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [10d91be] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 22:01:05,657 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [227224a7] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 22:01:05,666 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-4) [7b2c9b61] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:01:05,668 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [7b2c9b61] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 39a4acb5 >2013-03-07 22:01:05,679 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [7b2c9b61] FINISH, SetVdsStatusVDSCommand, log id: 39a4acb5 >2013-03-07 22:01:05,685 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-4) [7b2c9b61] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 22:01:05,705 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-4) [8b4e029] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:01:05,707 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-4) [6c49d26] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:01:05,709 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-4) [6c49d26] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 22:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [700a62ec] Autorecovering 1 hosts >2013-03-07 22:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [700a62ec] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [5c6e2f70] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [5c6e2f70] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [5c6e2f70] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1b64810 >2013-03-07 22:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [5c6e2f70] FINISH, SetVdsStatusVDSCommand, log id: 1b64810 >2013-03-07 22:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-83) [5c6e2f70] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 8c194d5 >2013-03-07 22:05:00,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [5c6e2f70] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 62e61e1d >2013-03-07 22:05:00,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [5c6e2f70] FINISH, GetHardwareInfoVDSCommand, log id: 62e61e1d >2013-03-07 22:05:00,451 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-83) [454568ec] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:05:00,454 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-83) [d783174] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:05:00,455 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-83) [d783174] FINISH, ActivateVdsVDSCommand, log id: 8c194d5 >2013-03-07 22:05:00,475 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [d783174] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:05:00,476 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-83) [d783174] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 22:05:00,477 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [d783174] Autorecovering 1 storage domains >2013-03-07 22:05:00,477 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [d783174] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 22:05:00,478 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [28be46c9] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:05:00,478 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [28be46c9] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 10:05 PM >2013-03-07 22:05:00,481 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [28be46c9] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 10:05 PM >2013-03-07 22:05:02,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [55a7a74b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3714c3e1 >2013-03-07 22:05:02,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [55a7a74b] FINISH, GetHardwareInfoVDSCommand, log id: 3714c3e1 >2013-03-07 22:05:02,952 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-59) [15429beb] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 22:05:02,955 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-59) [45ef10a8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 22:05:02,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5040c0b9 >2013-03-07 22:06:07,961 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 22:06:07,961 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:06:07,961 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 22:06:07,961 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 5040c0b9 >2013-03-07 22:06:07,965 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [4bc2cb93] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:06:07,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [4bc2cb93] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 62503e42 >2013-03-07 22:06:07,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [4bc2cb93] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 22:06:07,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [4bc2cb93] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 62503e42 >2013-03-07 22:06:07,985 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [45ef10a8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 22:06:07,995 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-59) [43438ce3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:06:07,996 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [43438ce3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 42de9091 >2013-03-07 22:06:07,998 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [43438ce3] FINISH, SetVdsStatusVDSCommand, log id: 42de9091 >2013-03-07 22:06:08,005 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [43438ce3] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 22:06:08,035 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [4999f1e1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:06:08,038 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [37d30159] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:06:08,039 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-59) [37d30159] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 22:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [598a4655] Autorecovering 1 hosts >2013-03-07 22:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [598a4655] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [5c1ad4be] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [5c1ad4be] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [5c1ad4be] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2e433d5b >2013-03-07 22:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [5c1ad4be] FINISH, SetVdsStatusVDSCommand, log id: 2e433d5b >2013-03-07 22:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [5c1ad4be] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5234ef05 >2013-03-07 22:10:00,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [5c1ad4be] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 61a6cc64 >2013-03-07 22:10:00,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [5c1ad4be] FINISH, GetHardwareInfoVDSCommand, log id: 61a6cc64 >2013-03-07 22:10:00,094 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-16) [121ef510] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:10:00,096 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-16) [24a5a91e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:10:00,097 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [24a5a91e] FINISH, ActivateVdsVDSCommand, log id: 5234ef05 >2013-03-07 22:10:00,117 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [24a5a91e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:10:00,118 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-16) [24a5a91e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 22:10:00,119 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [24a5a91e] Autorecovering 1 storage domains >2013-03-07 22:10:00,119 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [24a5a91e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 22:10:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [68e6cce3] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:10:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [68e6cce3] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 10:10 PM >2013-03-07 22:10:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [68e6cce3] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 10:10 PM >2013-03-07 22:10:02,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [22997f1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: ec8ead3 >2013-03-07 22:10:02,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [22997f1] FINISH, GetHardwareInfoVDSCommand, log id: ec8ead3 >2013-03-07 22:10:02,286 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-27) [766d541e] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 22:10:02,289 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-27) [17c0e4bb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 22:10:02,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 70332490 >2013-03-07 22:11:07,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 22:11:07,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:11:07,283 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 22:11:07,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 70332490 >2013-03-07 22:11:07,287 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [690285f0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:11:07,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [690285f0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 70ef3ed8 >2013-03-07 22:11:07,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [690285f0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 22:11:07,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [690285f0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 70ef3ed8 >2013-03-07 22:11:07,304 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [690285f0] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 22:11:07,306 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [17c0e4bb] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 22:11:07,314 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-27) [4fa520bd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:11:07,316 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [4fa520bd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 77daf30d >2013-03-07 22:11:07,317 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [4fa520bd] FINISH, SetVdsStatusVDSCommand, log id: 77daf30d >2013-03-07 22:11:07,329 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-27) [4fa520bd] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 22:11:07,353 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-27) [70f50f4f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:11:07,355 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-27) [2559972f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:11:07,357 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-27) [2559972f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 22:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [5812b250] Autorecovering 1 hosts >2013-03-07 22:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [5812b250] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [639e7319] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [639e7319] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [639e7319] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6954f72e >2013-03-07 22:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [639e7319] FINISH, SetVdsStatusVDSCommand, log id: 6954f72e >2013-03-07 22:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-24) [639e7319] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 294cc8b5 >2013-03-07 22:15:00,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [639e7319] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2734d481 >2013-03-07 22:15:00,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [639e7319] FINISH, GetHardwareInfoVDSCommand, log id: 2734d481 >2013-03-07 22:15:00,353 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-24) [632cfe40] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:15:00,355 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-24) [39e2d1b9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:15:00,356 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-24) [39e2d1b9] FINISH, ActivateVdsVDSCommand, log id: 294cc8b5 >2013-03-07 22:15:00,376 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [39e2d1b9] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:15:00,377 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-24) [39e2d1b9] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 22:15:00,378 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [39e2d1b9] Autorecovering 1 storage domains >2013-03-07 22:15:00,378 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [39e2d1b9] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 22:15:00,379 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [5bbb7ef8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:15:00,379 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [5bbb7ef8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 10:15 PM >2013-03-07 22:15:00,382 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [5bbb7ef8] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 10:15 PM >2013-03-07 22:15:01,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [27e3fb5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1e901342 >2013-03-07 22:15:01,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [27e3fb5] FINISH, GetHardwareInfoVDSCommand, log id: 1e901342 >2013-03-07 22:15:01,597 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-34) [578f93bb] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 22:15:01,601 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-34) [5d45e9b9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 22:15:01,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4fa7d855 >2013-03-07 22:16:05,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 22:16:05,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:16:05,892 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 22:16:05,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 4fa7d855 >2013-03-07 22:16:05,896 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [59b42bae] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:16:05,917 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [59b42bae] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4fb6bfe3 >2013-03-07 22:16:05,917 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [59b42bae] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 22:16:05,917 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [59b42bae] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4fb6bfe3 >2013-03-07 22:16:05,919 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [59b42bae] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 22:16:05,919 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [5d45e9b9] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 22:16:05,929 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-34) [1ae97282] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:16:05,931 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [1ae97282] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 161438a4 >2013-03-07 22:16:05,932 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [1ae97282] FINISH, SetVdsStatusVDSCommand, log id: 161438a4 >2013-03-07 22:16:05,938 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-34) [1ae97282] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 22:16:05,958 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-34) [304c2278] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:16:05,960 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-34) [5c32a01d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:16:05,962 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) [5c32a01d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 22:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [7754de2e] Autorecovering 1 hosts >2013-03-07 22:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [7754de2e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [4441cab] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [4441cab] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [4441cab] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5b459b5c >2013-03-07 22:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [4441cab] FINISH, SetVdsStatusVDSCommand, log id: 5b459b5c >2013-03-07 22:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [4441cab] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 71b39ebb >2013-03-07 22:20:00,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [4441cab] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6d25451f >2013-03-07 22:20:00,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [4441cab] FINISH, GetHardwareInfoVDSCommand, log id: 6d25451f >2013-03-07 22:20:00,102 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [30e8ba50] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:20:00,104 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [4c8b178f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:20:00,106 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [4c8b178f] FINISH, ActivateVdsVDSCommand, log id: 71b39ebb >2013-03-07 22:20:00,125 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [4c8b178f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:20:00,126 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-52) [4c8b178f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 22:20:00,128 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [4c8b178f] Autorecovering 1 storage domains >2013-03-07 22:20:00,128 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [4c8b178f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 22:20:00,128 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [28417a3a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:20:00,129 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [28417a3a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 10:20 PM >2013-03-07 22:20:00,132 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [28417a3a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 10:20 PM >2013-03-07 22:20:00,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 132e2cfe >2013-03-07 22:20:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) FINISH, GetHardwareInfoVDSCommand, log id: 132e2cfe >2013-03-07 22:20:00,204 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-73) [4920b20] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 22:20:00,207 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-73) [14b844eb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 22:20:00,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2e2f6699 >2013-03-07 22:21:06,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 22:21:06,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:21:06,255 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 22:21:06,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 2e2f6699 >2013-03-07 22:21:06,259 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [38303fc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:21:06,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [38303fc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5e3f048e >2013-03-07 22:21:06,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [38303fc] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 22:21:06,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [38303fc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5e3f048e >2013-03-07 22:21:06,276 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [38303fc] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 22:21:06,276 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [14b844eb] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 22:21:06,286 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-73) [2f16db65] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:21:06,287 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [2f16db65] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 36ed55a2 >2013-03-07 22:21:06,289 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [2f16db65] FINISH, SetVdsStatusVDSCommand, log id: 36ed55a2 >2013-03-07 22:21:06,295 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-73) [2f16db65] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 22:21:06,315 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-73) [35a11d54] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:21:06,318 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-73) [11a87ed8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:21:06,319 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-73) [11a87ed8] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 22:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [7d798295] Autorecovering 1 hosts >2013-03-07 22:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [7d798295] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [10cf10da] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [10cf10da] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-29) [10cf10da] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6e565ed >2013-03-07 22:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-29) [10cf10da] FINISH, SetVdsStatusVDSCommand, log id: 6e565ed >2013-03-07 22:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-29) [10cf10da] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5bf55174 >2013-03-07 22:25:00,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-29) [10cf10da] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4c406da5 >2013-03-07 22:25:00,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-29) [10cf10da] FINISH, GetHardwareInfoVDSCommand, log id: 4c406da5 >2013-03-07 22:25:00,273 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-29) [13a9382a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:25:00,276 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-29) [510bb28d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:25:00,277 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-29) [510bb28d] FINISH, ActivateVdsVDSCommand, log id: 5bf55174 >2013-03-07 22:25:00,297 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [510bb28d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:25:00,298 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-29) [510bb28d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 22:25:00,299 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [510bb28d] Autorecovering 1 storage domains >2013-03-07 22:25:00,299 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [510bb28d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 22:25:00,300 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [75e7377b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:25:00,300 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [75e7377b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 10:25 PM >2013-03-07 22:25:00,303 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [75e7377b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 10:25 PM >2013-03-07 22:25:00,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [419059ec] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 17f1f8d0 >2013-03-07 22:25:00,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [419059ec] FINISH, GetHardwareInfoVDSCommand, log id: 17f1f8d0 >2013-03-07 22:25:00,563 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-67) [537811a0] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 22:25:00,566 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-67) [5f0edad9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 22:25:00,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3098d5b >2013-03-07 22:26:05,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 22:26:05,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:26:05,599 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 22:26:05,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 3098d5b >2013-03-07 22:26:05,603 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [35f2765f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:26:05,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [35f2765f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 278e1693 >2013-03-07 22:26:05,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [35f2765f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 22:26:05,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [35f2765f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 278e1693 >2013-03-07 22:26:05,620 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [35f2765f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 22:26:05,620 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [5f0edad9] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 22:26:05,631 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-67) [20eca23b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:26:05,632 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [20eca23b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 137e2017 >2013-03-07 22:26:05,633 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [20eca23b] FINISH, SetVdsStatusVDSCommand, log id: 137e2017 >2013-03-07 22:26:05,640 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-67) [20eca23b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 22:26:05,660 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-67) [1a5eb8e5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:26:05,663 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-67) [36575662] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:26:05,664 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-67) [36575662] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 22:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) Autorecovering 1 hosts >2013-03-07 22:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-74) [e9d367a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-74) [e9d367a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [e9d367a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 45a35b76 >2013-03-07 22:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [e9d367a] FINISH, SetVdsStatusVDSCommand, log id: 45a35b76 >2013-03-07 22:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-74) [e9d367a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2d55df4d >2013-03-07 22:30:00,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [e9d367a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 17205bc9 >2013-03-07 22:30:00,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [e9d367a] FINISH, GetHardwareInfoVDSCommand, log id: 17205bc9 >2013-03-07 22:30:00,458 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-74) [61cedc45] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:30:00,461 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-74) [37c200f9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:30:00,462 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-74) [37c200f9] FINISH, ActivateVdsVDSCommand, log id: 2d55df4d >2013-03-07 22:30:00,482 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-74) [37c200f9] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:30:00,483 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-74) [37c200f9] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 22:30:00,484 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [37c200f9] Autorecovering 1 storage domains >2013-03-07 22:30:00,484 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [37c200f9] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 22:30:00,485 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-74) [535ce846] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:30:00,485 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-74) [535ce846] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 10:30 PM >2013-03-07 22:30:00,488 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-74) [535ce846] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 10:30 PM >2013-03-07 22:30:02,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [72589424] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: a6d29d2 >2013-03-07 22:30:02,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [72589424] FINISH, GetHardwareInfoVDSCommand, log id: a6d29d2 >2013-03-07 22:30:02,922 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-2) [6278d40] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 22:30:02,925 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-2) [4b363410] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 22:30:02,935 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 43bc5b87 >2013-03-07 22:30:33,709 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-59) [37d30159] Start refreshing all users data >2013-03-07 22:30:34,613 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-46) [497a2e37] Attempting to update VMs/Templates Ovf. >2013-03-07 22:30:34,614 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-46) [497a2e37] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 22:30:34,615 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-46) [497a2e37] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 22:30:34,615 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-46) [497a2e37] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 22:30:34,616 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-46) [497a2e37] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 22:30:34,616 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-46) [497a2e37] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 22:30:34,617 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-46) [497a2e37] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 22:31:06,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 22:31:06,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:31:06,904 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 22:31:06,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 43bc5b87 >2013-03-07 22:31:06,908 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [13085fc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:31:06,929 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [13085fc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: fbf017a >2013-03-07 22:31:06,929 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [13085fc] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 22:31:06,929 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [13085fc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: fbf017a >2013-03-07 22:31:06,932 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [13085fc] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 22:31:06,932 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [4b363410] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 22:31:06,942 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-2) [c53d851] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:31:06,944 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [c53d851] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5f229a57 >2013-03-07 22:31:06,945 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [c53d851] FINISH, SetVdsStatusVDSCommand, log id: 5f229a57 >2013-03-07 22:31:06,950 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [c53d851] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 22:31:06,980 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [604004ca] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:31:06,983 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [59868ca0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:31:06,984 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-2) [59868ca0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 22:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [1ed6de3e] Autorecovering 1 hosts >2013-03-07 22:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [1ed6de3e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [71afb78e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [71afb78e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [71afb78e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 31fbe425 >2013-03-07 22:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [71afb78e] FINISH, SetVdsStatusVDSCommand, log id: 31fbe425 >2013-03-07 22:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [71afb78e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 75fb35af >2013-03-07 22:35:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [71afb78e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7851b31f >2013-03-07 22:35:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [71afb78e] FINISH, GetHardwareInfoVDSCommand, log id: 7851b31f >2013-03-07 22:35:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [755db971] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:35:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [7def33] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:35:00,093 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [7def33] FINISH, ActivateVdsVDSCommand, log id: 75fb35af >2013-03-07 22:35:00,113 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [7def33] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:35:00,114 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-94) [7def33] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 22:35:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [7def33] Autorecovering 1 storage domains >2013-03-07 22:35:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [7def33] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 22:35:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [1b0a3a8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:35:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [1b0a3a8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 10:35 PM >2013-03-07 22:35:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [1b0a3a8] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 10:35 PM >2013-03-07 22:35:01,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [46564268] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2900aeed >2013-03-07 22:35:01,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [46564268] FINISH, GetHardwareInfoVDSCommand, log id: 2900aeed >2013-03-07 22:35:01,227 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-58) [1aaca47] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 22:35:01,230 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-58) [317612d5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 22:35:01,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: c990c51 >2013-03-07 22:36:06,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 22:36:06,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:36:06,210 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 22:36:06,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: c990c51 >2013-03-07 22:36:06,215 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [7de14837] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:36:06,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [7de14837] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 64f31f73 >2013-03-07 22:36:06,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [7de14837] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 22:36:06,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [7de14837] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 64f31f73 >2013-03-07 22:36:06,232 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [7de14837] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 22:36:06,233 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [317612d5] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 22:36:06,242 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-58) [38bf8954] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:36:06,244 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [38bf8954] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3ffedb97 >2013-03-07 22:36:06,245 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [38bf8954] FINISH, SetVdsStatusVDSCommand, log id: 3ffedb97 >2013-03-07 22:36:06,250 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-58) [38bf8954] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 22:36:06,275 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-58) [783724f9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:36:06,277 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-58) [364f59a5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:36:06,279 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-58) [364f59a5] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 22:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [22438700] Autorecovering 1 hosts >2013-03-07 22:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [22438700] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-5) [7be4f36a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-5) [7be4f36a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [7be4f36a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 38c3aca6 >2013-03-07 22:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [7be4f36a] FINISH, SetVdsStatusVDSCommand, log id: 38c3aca6 >2013-03-07 22:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-5) [7be4f36a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 195bfa4f >2013-03-07 22:40:00,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [7be4f36a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6429679e >2013-03-07 22:40:00,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [7be4f36a] FINISH, GetHardwareInfoVDSCommand, log id: 6429679e >2013-03-07 22:40:00,561 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-5) [1f1512e8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:40:00,563 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-5) [1226f36a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:40:00,565 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-5) [1226f36a] FINISH, ActivateVdsVDSCommand, log id: 195bfa4f >2013-03-07 22:40:00,584 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-5) [1226f36a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:40:00,585 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-5) [1226f36a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 22:40:00,587 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [1226f36a] Autorecovering 1 storage domains >2013-03-07 22:40:00,587 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [1226f36a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 22:40:00,587 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-5) [434cb3a8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:40:00,588 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-5) [434cb3a8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 10:40 PM >2013-03-07 22:40:00,590 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-5) [434cb3a8] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 10:40 PM >2013-03-07 22:40:00,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2f65ca5a >2013-03-07 22:40:00,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) FINISH, GetHardwareInfoVDSCommand, log id: 2f65ca5a >2013-03-07 22:40:00,668 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-69) [165a00f6] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 22:40:00,671 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-69) [29a73c61] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 22:40:00,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6f2d729a >2013-03-07 22:41:05,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 22:41:05,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:41:05,663 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 22:41:05,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 6f2d729a >2013-03-07 22:41:05,666 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [20fc55bd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:41:05,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [20fc55bd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 269336ff >2013-03-07 22:41:05,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [20fc55bd] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 22:41:05,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [20fc55bd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 269336ff >2013-03-07 22:41:05,684 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [20fc55bd] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 22:41:05,684 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [29a73c61] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 22:41:05,694 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-69) [2f46f0f1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:41:05,695 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [2f46f0f1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3fdcfd02 >2013-03-07 22:41:05,697 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [2f46f0f1] FINISH, SetVdsStatusVDSCommand, log id: 3fdcfd02 >2013-03-07 22:41:05,703 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-69) [2f46f0f1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 22:41:05,723 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-69) [4ada46fb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:41:05,725 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-69) [576edfad] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:41:05,726 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-69) [576edfad] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 22:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) Autorecovering 1 hosts >2013-03-07 22:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:45:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [7e3a0c72] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [7e3a0c72] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [7e3a0c72] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2fb01eef >2013-03-07 22:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [7e3a0c72] FINISH, SetVdsStatusVDSCommand, log id: 2fb01eef >2013-03-07 22:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-21) [7e3a0c72] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7c239223 >2013-03-07 22:45:00,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [7e3a0c72] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 61e1ece4 >2013-03-07 22:45:00,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [7e3a0c72] FINISH, GetHardwareInfoVDSCommand, log id: 61e1ece4 >2013-03-07 22:45:00,087 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-21) [3db6839c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:45:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-21) [2c817e1d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:45:00,091 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-21) [2c817e1d] FINISH, ActivateVdsVDSCommand, log id: 7c239223 >2013-03-07 22:45:00,110 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [2c817e1d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:45:00,111 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-21) [2c817e1d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 22:45:00,112 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [2c817e1d] Autorecovering 1 storage domains >2013-03-07 22:45:00,112 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [2c817e1d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 22:45:00,113 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [2b03ae05] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:45:00,113 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [2b03ae05] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 10:45 PM >2013-03-07 22:45:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [2b03ae05] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 10:45 PM >2013-03-07 22:45:02,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [493d2a36] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 103af3ea >2013-03-07 22:45:02,935 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [493d2a36] FINISH, GetHardwareInfoVDSCommand, log id: 103af3ea >2013-03-07 22:45:02,972 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-22) [5cca06db] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 22:45:02,980 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-22) [53df954c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 22:45:02,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 61808dab >2013-03-07 22:46:05,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 22:46:05,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:46:05,954 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 22:46:05,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 61808dab >2013-03-07 22:46:05,958 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [5590b05a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:46:05,973 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [5590b05a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 258a67e4 >2013-03-07 22:46:05,973 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [5590b05a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 22:46:05,973 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [5590b05a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 258a67e4 >2013-03-07 22:46:05,975 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [5590b05a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 22:46:05,975 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [53df954c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 22:46:05,986 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-22) [172b6e7e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:46:05,987 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [172b6e7e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 372faf9d >2013-03-07 22:46:05,988 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [172b6e7e] FINISH, SetVdsStatusVDSCommand, log id: 372faf9d >2013-03-07 22:46:05,997 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [172b6e7e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 22:46:06,018 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [55af3947] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:46:06,021 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [9253b4f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:46:06,022 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-22) [9253b4f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 22:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-66) [7b0772eb] Autorecovering 1 hosts >2013-03-07 22:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-66) [7b0772eb] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-66) [2167e9f8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-66) [2167e9f8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [2167e9f8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 784cfde >2013-03-07 22:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [2167e9f8] FINISH, SetVdsStatusVDSCommand, log id: 784cfde >2013-03-07 22:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-66) [2167e9f8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 313feb54 >2013-03-07 22:50:00,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [2167e9f8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 73b882f3 >2013-03-07 22:50:00,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [2167e9f8] FINISH, GetHardwareInfoVDSCommand, log id: 73b882f3 >2013-03-07 22:50:00,097 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-66) [36d5b0a6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:50:00,100 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-66) [22316c6c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:50:00,101 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-66) [22316c6c] FINISH, ActivateVdsVDSCommand, log id: 313feb54 >2013-03-07 22:50:00,120 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-66) [22316c6c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:50:00,121 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-66) [22316c6c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 22:50:00,122 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-66) [22316c6c] Autorecovering 1 storage domains >2013-03-07 22:50:00,122 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-66) [22316c6c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 22:50:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-66) [44b5caad] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:50:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-66) [44b5caad] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 10:50 PM >2013-03-07 22:50:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-66) [44b5caad] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 10:50 PM >2013-03-07 22:50:00,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [5f60280d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 8e3b5b7 >2013-03-07 22:50:00,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [5f60280d] FINISH, GetHardwareInfoVDSCommand, log id: 8e3b5b7 >2013-03-07 22:50:00,270 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-63) [489a7e5f] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 22:50:00,273 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-63) [29f1df25] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 22:50:00,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 389e95dd >2013-03-07 22:51:05,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 22:51:05,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:51:05,314 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 22:51:05,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 389e95dd >2013-03-07 22:51:05,317 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [4aa9f414] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:51:05,331 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [4aa9f414] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 14fbfd76 >2013-03-07 22:51:05,331 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [4aa9f414] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 22:51:05,331 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [4aa9f414] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 14fbfd76 >2013-03-07 22:51:05,334 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [4aa9f414] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 22:51:05,335 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [29f1df25] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 22:51:05,344 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-63) [59f33109] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:51:05,346 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [59f33109] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3398a3da >2013-03-07 22:51:05,347 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [59f33109] FINISH, SetVdsStatusVDSCommand, log id: 3398a3da >2013-03-07 22:51:05,353 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-63) [59f33109] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 22:51:05,376 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-63) [4ac3ba20] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:51:05,379 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-63) [5c7fd00e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:51:05,380 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-63) [5c7fd00e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 22:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [3e2c6ba7] Autorecovering 1 hosts >2013-03-07 22:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [3e2c6ba7] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [65199c61] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [65199c61] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [65199c61] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7bd1ebc2 >2013-03-07 22:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [65199c61] FINISH, SetVdsStatusVDSCommand, log id: 7bd1ebc2 >2013-03-07 22:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-44) [65199c61] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 42cf75e2 >2013-03-07 22:55:00,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [65199c61] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1945faa2 >2013-03-07 22:55:00,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [65199c61] FINISH, GetHardwareInfoVDSCommand, log id: 1945faa2 >2013-03-07 22:55:00,578 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-44) [1efd0296] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:55:00,580 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-44) [58420f8c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:55:00,586 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-44) [58420f8c] FINISH, ActivateVdsVDSCommand, log id: 42cf75e2 >2013-03-07 22:55:00,605 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [58420f8c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 22:55:00,606 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-44) [58420f8c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 22:55:00,608 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [58420f8c] Autorecovering 1 storage domains >2013-03-07 22:55:00,608 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [58420f8c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 22:55:00,608 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [38bfed06] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:55:00,609 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [38bfed06] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 10:55 PM >2013-03-07 22:55:00,611 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [38bfed06] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 10:55 PM >2013-03-07 22:55:02,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [37e7b9cb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5c26eebc >2013-03-07 22:55:02,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [37e7b9cb] FINISH, GetHardwareInfoVDSCommand, log id: 5c26eebc >2013-03-07 22:55:02,628 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-11) [5d3b1706] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 22:55:02,631 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-11) [33887afa] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 22:55:02,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 297fab7 >2013-03-07 22:56:06,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 22:56:06,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 22:56:06,648 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 22:56:06,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 297fab7 >2013-03-07 22:56:06,652 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [6feb3835] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 22:56:06,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [6feb3835] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7a7a2048 >2013-03-07 22:56:06,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [6feb3835] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 22:56:06,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [6feb3835] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7a7a2048 >2013-03-07 22:56:06,676 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [6feb3835] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 22:56:06,676 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [33887afa] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 22:56:06,686 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-11) [2756b55a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:56:06,688 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [2756b55a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4fa8643 >2013-03-07 22:56:06,689 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [2756b55a] FINISH, SetVdsStatusVDSCommand, log id: 4fa8643 >2013-03-07 22:56:06,696 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-11) [2756b55a] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 22:56:06,721 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-11) [48b1f53e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:56:06,724 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-11) [283308f8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 22:56:06,725 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-11) [283308f8] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 23:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) Autorecovering 1 hosts >2013-03-07 23:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [272fdb6c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [272fdb6c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [272fdb6c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7521f2ed >2013-03-07 23:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [272fdb6c] FINISH, SetVdsStatusVDSCommand, log id: 7521f2ed >2013-03-07 23:00:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-57) [272fdb6c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 146fdad8 >2013-03-07 23:00:00,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [272fdb6c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6ef8e8b3 >2013-03-07 23:00:00,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [272fdb6c] FINISH, GetHardwareInfoVDSCommand, log id: 6ef8e8b3 >2013-03-07 23:00:00,094 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-57) [786897a1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:00:00,097 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-57) [2c4a6f88] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:00:00,098 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-57) [2c4a6f88] FINISH, ActivateVdsVDSCommand, log id: 146fdad8 >2013-03-07 23:00:00,117 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [2c4a6f88] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:00:00,118 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-57) [2c4a6f88] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 23:00:00,120 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [2c4a6f88] Autorecovering 1 storage domains >2013-03-07 23:00:00,120 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [2c4a6f88] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 23:00:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [225c2169] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:00:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [225c2169] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 11:00 PM >2013-03-07 23:00:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [225c2169] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 11:00 PM >2013-03-07 23:00:00,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [30d9a407] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6c39bf67 >2013-03-07 23:00:00,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [30d9a407] FINISH, GetHardwareInfoVDSCommand, log id: 6c39bf67 >2013-03-07 23:00:00,966 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-26) [7ba7b203] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 23:00:00,969 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-26) [9b6079c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 23:00:00,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 65a21e9e >2013-03-07 23:01:04,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 23:01:04,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:01:04,968 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 23:01:04,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 65a21e9e >2013-03-07 23:01:04,972 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [6102791f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:01:04,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [6102791f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6b4f6306 >2013-03-07 23:01:04,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [6102791f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 23:01:04,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [6102791f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6b4f6306 >2013-03-07 23:01:04,988 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [6102791f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 23:01:04,989 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [9b6079c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 23:01:04,998 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-26) [67f059fb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:01:05,000 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [67f059fb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: ea0b27a >2013-03-07 23:01:05,001 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [67f059fb] FINISH, SetVdsStatusVDSCommand, log id: ea0b27a >2013-03-07 23:01:05,007 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [67f059fb] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 23:01:05,049 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [63e55041] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:01:05,052 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [74ce696f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:01:05,053 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-26) [74ce696f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 23:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [7d9f4ed3] Autorecovering 1 hosts >2013-03-07 23:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [7d9f4ed3] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [593f4f51] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [593f4f51] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [593f4f51] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4c256d37 >2013-03-07 23:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [593f4f51] FINISH, SetVdsStatusVDSCommand, log id: 4c256d37 >2013-03-07 23:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [593f4f51] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2b38c73f >2013-03-07 23:05:00,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [593f4f51] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3cd10 >2013-03-07 23:05:00,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [593f4f51] FINISH, GetHardwareInfoVDSCommand, log id: 3cd10 >2013-03-07 23:05:00,765 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-45) [75239268] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:05:00,768 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-45) [54a65f09] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:05:00,769 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [54a65f09] FINISH, ActivateVdsVDSCommand, log id: 2b38c73f >2013-03-07 23:05:00,788 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [54a65f09] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:05:00,789 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-45) [54a65f09] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 23:05:00,791 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [54a65f09] Autorecovering 1 storage domains >2013-03-07 23:05:00,791 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [54a65f09] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 23:05:00,791 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [727e03e5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:05:00,791 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [727e03e5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 11:05 PM >2013-03-07 23:05:00,794 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [727e03e5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 11:05 PM >2013-03-07 23:05:02,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [497a2e37] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1de2d383 >2013-03-07 23:05:02,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [497a2e37] FINISH, GetHardwareInfoVDSCommand, log id: 1de2d383 >2013-03-07 23:05:02,297 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-46) [7a3d0f73] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 23:05:02,300 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-46) [2e54ea0b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 23:05:02,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7633f13d >2013-03-07 23:06:06,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 23:06:06,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:06:06,320 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 23:06:06,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 7633f13d >2013-03-07 23:06:06,324 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [639a4da8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:06:06,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [639a4da8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5f45b25c >2013-03-07 23:06:06,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [639a4da8] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 23:06:06,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [639a4da8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5f45b25c >2013-03-07 23:06:06,355 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [639a4da8] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 23:06:06,358 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [2e54ea0b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 23:06:06,365 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-46) [7fc975a4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:06:06,367 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [7fc975a4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 368fe3aa >2013-03-07 23:06:06,368 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [7fc975a4] FINISH, SetVdsStatusVDSCommand, log id: 368fe3aa >2013-03-07 23:06:06,374 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-46) [7fc975a4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 23:06:06,394 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-46) [6b01b909] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:06:06,397 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-46) [4c1eb652] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:06:06,398 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-46) [4c1eb652] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 23:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [3a13c4c1] Autorecovering 1 hosts >2013-03-07 23:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [3a13c4c1] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-82) [361696dc] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-82) [361696dc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [361696dc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 39486a89 >2013-03-07 23:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [361696dc] FINISH, SetVdsStatusVDSCommand, log id: 39486a89 >2013-03-07 23:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-82) [361696dc] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3b05b97a >2013-03-07 23:10:00,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [361696dc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 16a1c167 >2013-03-07 23:10:00,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [361696dc] FINISH, GetHardwareInfoVDSCommand, log id: 16a1c167 >2013-03-07 23:10:00,094 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-82) [11547210] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:10:00,096 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-82) [2eb0edf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:10:00,097 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-82) [2eb0edf] FINISH, ActivateVdsVDSCommand, log id: 3b05b97a >2013-03-07 23:10:00,117 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-82) [2eb0edf] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:10:00,118 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-82) [2eb0edf] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 23:10:00,119 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [2eb0edf] Autorecovering 1 storage domains >2013-03-07 23:10:00,119 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [2eb0edf] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 23:10:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-82) [66e3f062] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:10:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-82) [66e3f062] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 11:10 PM >2013-03-07 23:10:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-82) [66e3f062] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 11:10 PM >2013-03-07 23:10:00,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [6761b250] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 44a4d84b >2013-03-07 23:10:00,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [6761b250] FINISH, GetHardwareInfoVDSCommand, log id: 44a4d84b >2013-03-07 23:10:00,644 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-54) [76f03464] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 23:10:00,648 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-54) [50eb4925] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 23:10:00,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1a056da2 >2013-03-07 23:11:05,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 23:11:05,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:11:05,639 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 23:11:05,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 1a056da2 >2013-03-07 23:11:05,643 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [34646fc1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:11:05,658 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [34646fc1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 55baeeca >2013-03-07 23:11:05,658 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [34646fc1] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 23:11:05,658 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [34646fc1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 55baeeca >2013-03-07 23:11:05,663 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [50eb4925] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 23:11:05,673 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-54) [3934efb2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:11:05,674 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [3934efb2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3bf3a309 >2013-03-07 23:11:05,675 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [3934efb2] FINISH, SetVdsStatusVDSCommand, log id: 3bf3a309 >2013-03-07 23:11:05,682 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [3934efb2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 23:11:05,702 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [10ab88a0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:11:05,705 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [4ae1c7cf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:11:05,706 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-54) [4ae1c7cf] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 23:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [722e1bc0] Autorecovering 1 hosts >2013-03-07 23:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [722e1bc0] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [71d95fe] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [71d95fe] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [71d95fe] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5ab61711 >2013-03-07 23:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [71d95fe] FINISH, SetVdsStatusVDSCommand, log id: 5ab61711 >2013-03-07 23:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-78) [71d95fe] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 631ecb97 >2013-03-07 23:15:00,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [71d95fe] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 57289992 >2013-03-07 23:15:00,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [71d95fe] FINISH, GetHardwareInfoVDSCommand, log id: 57289992 >2013-03-07 23:15:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-78) [2799af47] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:15:00,095 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-78) [61cccad] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:15:00,096 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-78) [61cccad] FINISH, ActivateVdsVDSCommand, log id: 631ecb97 >2013-03-07 23:15:00,115 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [61cccad] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:15:00,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-78) [61cccad] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 23:15:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [61cccad] Autorecovering 1 storage domains >2013-03-07 23:15:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [61cccad] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 23:15:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [43661eaa] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:15:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [43661eaa] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 11:15 PM >2013-03-07 23:15:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [43661eaa] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 11:15 PM >2013-03-07 23:15:02,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [6ad3d54c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 199cfb78 >2013-03-07 23:15:02,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [6ad3d54c] FINISH, GetHardwareInfoVDSCommand, log id: 199cfb78 >2013-03-07 23:15:02,958 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-33) [54e0e5f0] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 23:15:02,961 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-33) [6b0ad9ae] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 23:15:02,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2fc66ad7 >2013-03-07 23:16:08,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 23:16:08,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:16:08,008 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 23:16:08,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 2fc66ad7 >2013-03-07 23:16:08,012 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [42164180] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:16:08,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [42164180] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3dc3708b >2013-03-07 23:16:08,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [42164180] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 23:16:08,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [42164180] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3dc3708b >2013-03-07 23:16:08,029 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [42164180] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 23:16:08,029 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [6b0ad9ae] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 23:16:08,039 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-33) [3f686624] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:16:08,040 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [3f686624] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 75daca92 >2013-03-07 23:16:08,042 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [3f686624] FINISH, SetVdsStatusVDSCommand, log id: 75daca92 >2013-03-07 23:16:08,048 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-33) [3f686624] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 23:16:08,075 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-33) [3da20465] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:16:08,077 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-33) [7b678968] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:16:08,079 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-33) [7b678968] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 23:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [4608f341] Autorecovering 1 hosts >2013-03-07 23:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [4608f341] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [49b392a3] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [49b392a3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [49b392a3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 15787313 >2013-03-07 23:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [49b392a3] FINISH, SetVdsStatusVDSCommand, log id: 15787313 >2013-03-07 23:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-88) [49b392a3] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7fb777ae >2013-03-07 23:20:00,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [49b392a3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3baf8768 >2013-03-07 23:20:00,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [49b392a3] FINISH, GetHardwareInfoVDSCommand, log id: 3baf8768 >2013-03-07 23:20:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-88) [5ae8d403] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:20:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-88) [7bae10f4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:20:00,095 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-88) [7bae10f4] FINISH, ActivateVdsVDSCommand, log id: 7fb777ae >2013-03-07 23:20:00,114 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [7bae10f4] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:20:00,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-88) [7bae10f4] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 23:20:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [7bae10f4] Autorecovering 1 storage domains >2013-03-07 23:20:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [7bae10f4] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 23:20:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [7e412d47] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:20:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [7e412d47] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 11:20 PM >2013-03-07 23:20:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [7e412d47] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 11:20 PM >2013-03-07 23:20:02,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [5ccf95a5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3c938cbf >2013-03-07 23:20:02,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [5ccf95a5] FINISH, GetHardwareInfoVDSCommand, log id: 3c938cbf >2013-03-07 23:20:02,349 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-50) [32075f45] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 23:20:02,352 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-50) [e8bc99e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 23:20:02,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4fadcf65 >2013-03-07 23:21:08,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 23:21:08,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:21:08,409 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 23:21:08,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 4fadcf65 >2013-03-07 23:21:08,413 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [3c2397a7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:21:08,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [3c2397a7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 43d2e7ef >2013-03-07 23:21:08,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [3c2397a7] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 23:21:08,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [3c2397a7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 43d2e7ef >2013-03-07 23:21:08,430 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [3c2397a7] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 23:21:08,430 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [e8bc99e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 23:21:08,440 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-50) [aa2cec2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:21:08,442 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [aa2cec2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2f000dc3 >2013-03-07 23:21:08,443 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [aa2cec2] FINISH, SetVdsStatusVDSCommand, log id: 2f000dc3 >2013-03-07 23:21:08,450 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-50) [aa2cec2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 23:21:08,475 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-50) [51c83462] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:21:08,477 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-50) [75937e8e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:21:08,479 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-50) [75937e8e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 23:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [7f8f1391] Autorecovering 1 hosts >2013-03-07 23:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [7f8f1391] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [5d43f302] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [5d43f302] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [5d43f302] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4579ca53 >2013-03-07 23:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [5d43f302] FINISH, SetVdsStatusVDSCommand, log id: 4579ca53 >2013-03-07 23:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-89) [5d43f302] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 518314fb >2013-03-07 23:25:00,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [5d43f302] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6570da1f >2013-03-07 23:25:00,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [5d43f302] FINISH, GetHardwareInfoVDSCommand, log id: 6570da1f >2013-03-07 23:25:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-89) [a550e35] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:25:00,094 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-89) [688e69d1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:25:00,096 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-89) [688e69d1] FINISH, ActivateVdsVDSCommand, log id: 518314fb >2013-03-07 23:25:00,115 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [688e69d1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:25:00,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-89) [688e69d1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 23:25:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [688e69d1] Autorecovering 1 storage domains >2013-03-07 23:25:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [688e69d1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 23:25:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [f72379c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:25:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [f72379c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 11:25 PM >2013-03-07 23:25:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [f72379c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 11:25 PM >2013-03-07 23:25:02,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [53b9b3d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2ed9c2b0 >2013-03-07 23:25:02,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [53b9b3d] FINISH, GetHardwareInfoVDSCommand, log id: 2ed9c2b0 >2013-03-07 23:25:02,723 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-32) [1a7a6229] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 23:25:02,726 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-32) [541944cf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 23:25:02,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5639c922 >2013-03-07 23:26:07,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 23:26:07,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:26:07,734 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 23:26:07,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 5639c922 >2013-03-07 23:26:07,739 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [461fa733] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:26:07,753 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [461fa733] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3da8d6a6 >2013-03-07 23:26:07,753 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [461fa733] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 23:26:07,757 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [461fa733] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3da8d6a6 >2013-03-07 23:26:07,765 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [461fa733] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 23:26:07,765 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [541944cf] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 23:26:07,777 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-32) [55886d6d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:26:07,782 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [55886d6d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 35925c93 >2013-03-07 23:26:07,784 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [55886d6d] FINISH, SetVdsStatusVDSCommand, log id: 35925c93 >2013-03-07 23:26:07,789 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-32) [55886d6d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 23:26:07,810 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-32) [59c8ea44] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:26:07,812 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-32) [2510b209] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:26:07,814 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-32) [2510b209] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 23:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [25cefe9] Autorecovering 1 hosts >2013-03-07 23:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [25cefe9] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [73b68c1c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [73b68c1c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [73b68c1c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4ca8e562 >2013-03-07 23:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [73b68c1c] FINISH, SetVdsStatusVDSCommand, log id: 4ca8e562 >2013-03-07 23:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-77) [73b68c1c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: a34fbda >2013-03-07 23:30:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [73b68c1c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6e64de1f >2013-03-07 23:30:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [73b68c1c] FINISH, GetHardwareInfoVDSCommand, log id: 6e64de1f >2013-03-07 23:30:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-77) [186c39e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:30:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-77) [60bc2fb4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:30:00,094 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-77) [60bc2fb4] FINISH, ActivateVdsVDSCommand, log id: a34fbda >2013-03-07 23:30:00,113 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [60bc2fb4] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:30:00,114 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-77) [60bc2fb4] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 23:30:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [60bc2fb4] Autorecovering 1 storage domains >2013-03-07 23:30:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [60bc2fb4] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 23:30:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [530c4d5e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:30:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [530c4d5e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 11:30 PM >2013-03-07 23:30:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [530c4d5e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 11:30 PM >2013-03-07 23:30:02,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [59868ca0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 48309bac >2013-03-07 23:30:02,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [59868ca0] FINISH, GetHardwareInfoVDSCommand, log id: 48309bac >2013-03-07 23:30:02,065 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-2) [7743a444] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 23:30:02,069 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-2) [3079e8b8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 23:30:02,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 438fd215 >2013-03-07 23:30:33,710 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-68) [65873a9c] Start refreshing all users data >2013-03-07 23:30:34,617 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2385e91c] Attempting to update VMs/Templates Ovf. >2013-03-07 23:30:34,618 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2385e91c] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 23:30:34,619 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2385e91c] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 23:30:34,619 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2385e91c] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 23:30:34,620 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2385e91c] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 23:30:34,620 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2385e91c] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 23:30:34,620 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2385e91c] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-07 23:31:07,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 23:31:07,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:31:07,070 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 23:31:07,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 438fd215 >2013-03-07 23:31:07,073 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [16b2d82a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:31:07,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [16b2d82a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 48932b41 >2013-03-07 23:31:07,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [16b2d82a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 23:31:07,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [16b2d82a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 48932b41 >2013-03-07 23:31:07,091 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [16b2d82a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 23:31:07,092 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [3079e8b8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 23:31:07,103 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-2) [b9a59b6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:31:07,104 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [b9a59b6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4705bb98 >2013-03-07 23:31:07,105 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [b9a59b6] FINISH, SetVdsStatusVDSCommand, log id: 4705bb98 >2013-03-07 23:31:07,111 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [b9a59b6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 23:31:07,131 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [524dc454] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:31:07,134 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [679c2827] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:31:07,135 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-2) [679c2827] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 23:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [319f8114] Autorecovering 1 hosts >2013-03-07 23:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [319f8114] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [16f4044a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [16f4044a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [16f4044a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 60da87b3 >2013-03-07 23:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [16f4044a] FINISH, SetVdsStatusVDSCommand, log id: 60da87b3 >2013-03-07 23:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-90) [16f4044a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 667c27e >2013-03-07 23:35:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [16f4044a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 732538c7 >2013-03-07 23:35:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [16f4044a] FINISH, GetHardwareInfoVDSCommand, log id: 732538c7 >2013-03-07 23:35:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-90) [7bb34744] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:35:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-90) [720289b5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:35:00,095 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-90) [720289b5] FINISH, ActivateVdsVDSCommand, log id: 667c27e >2013-03-07 23:35:00,115 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [720289b5] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:35:00,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [720289b5] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 23:35:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [720289b5] Autorecovering 1 storage domains >2013-03-07 23:35:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [720289b5] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 23:35:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [1022b3a1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:35:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [1022b3a1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 11:35 PM >2013-03-07 23:35:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [1022b3a1] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 11:35 PM >2013-03-07 23:35:01,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [4ec30a2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 739f46df >2013-03-07 23:35:01,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [4ec30a2] FINISH, GetHardwareInfoVDSCommand, log id: 739f46df >2013-03-07 23:35:01,385 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-86) [442d8861] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 23:35:01,389 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-86) [ec412b8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 23:35:01,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 79be08f >2013-03-07 23:36:05,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 23:36:05,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:36:05,677 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 23:36:05,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 79be08f >2013-03-07 23:36:05,681 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [55e117e0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:36:05,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [55e117e0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1df44dca >2013-03-07 23:36:05,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [55e117e0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 23:36:05,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [55e117e0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1df44dca >2013-03-07 23:36:05,698 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [55e117e0] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 23:36:05,698 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [ec412b8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 23:36:05,708 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-86) [417dff2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:36:05,710 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [417dff2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7ee50da9 >2013-03-07 23:36:05,711 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [417dff2] FINISH, SetVdsStatusVDSCommand, log id: 7ee50da9 >2013-03-07 23:36:05,716 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-86) [417dff2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 23:36:05,736 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-86) [4d3e6f2e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:36:05,739 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-86) [10f43d4e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:36:05,740 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-86) [10f43d4e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 23:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [6cfcca05] Autorecovering 1 hosts >2013-03-07 23:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [6cfcca05] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-80) [29bcd8c7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-80) [29bcd8c7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [29bcd8c7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 44e386d >2013-03-07 23:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [29bcd8c7] FINISH, SetVdsStatusVDSCommand, log id: 44e386d >2013-03-07 23:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-80) [29bcd8c7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3861e9e6 >2013-03-07 23:40:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [29bcd8c7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 79f2fbb0 >2013-03-07 23:40:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [29bcd8c7] FINISH, GetHardwareInfoVDSCommand, log id: 79f2fbb0 >2013-03-07 23:40:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-80) [69ab3a33] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:40:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-80) [43ce2379] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:40:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-80) [43ce2379] FINISH, ActivateVdsVDSCommand, log id: 3861e9e6 >2013-03-07 23:40:00,111 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-80) [43ce2379] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:40:00,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-80) [43ce2379] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 23:40:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [43ce2379] Autorecovering 1 storage domains >2013-03-07 23:40:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [43ce2379] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 23:40:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-80) [44062586] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:40:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-80) [44062586] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 11:40 PM >2013-03-07 23:40:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-80) [44062586] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 11:40 PM >2013-03-07 23:40:02,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [530c4d5e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1730288c >2013-03-07 23:40:02,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [530c4d5e] FINISH, GetHardwareInfoVDSCommand, log id: 1730288c >2013-03-07 23:40:02,979 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-77) [4922e0f6] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 23:40:02,983 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-77) [29fc0b38] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 23:40:02,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3eeb196f >2013-03-07 23:41:06,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 23:41:06,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:41:06,972 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 23:41:06,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 3eeb196f >2013-03-07 23:41:06,976 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [69d33d69] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:41:06,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [69d33d69] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7e27b2eb >2013-03-07 23:41:06,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [69d33d69] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 23:41:06,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [69d33d69] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7e27b2eb >2013-03-07 23:41:06,993 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [69d33d69] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 23:41:06,993 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [29fc0b38] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 23:41:07,003 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-77) [e7525e4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:41:07,004 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [e7525e4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 430c58e2 >2013-03-07 23:41:07,005 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [e7525e4] FINISH, SetVdsStatusVDSCommand, log id: 430c58e2 >2013-03-07 23:41:07,012 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-77) [e7525e4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 23:41:07,032 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-77) [162ea762] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:41:07,034 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-77) [68207617] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:41:07,035 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-77) [68207617] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 23:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) Autorecovering 1 hosts >2013-03-07 23:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-85) [7490b65d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-85) [7490b65d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-85) [7490b65d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7e26f0d3 >2013-03-07 23:45:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-85) [7490b65d] FINISH, SetVdsStatusVDSCommand, log id: 7e26f0d3 >2013-03-07 23:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-85) [7490b65d] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2f53c26c >2013-03-07 23:45:00,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-85) [7490b65d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1e329cf8 >2013-03-07 23:45:00,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-85) [7490b65d] FINISH, GetHardwareInfoVDSCommand, log id: 1e329cf8 >2013-03-07 23:45:00,465 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-85) [780c9d59] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:45:00,467 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-85) [c29d518] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:45:00,469 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-85) [c29d518] FINISH, ActivateVdsVDSCommand, log id: 2f53c26c >2013-03-07 23:45:00,488 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-85) [c29d518] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:45:00,489 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-85) [c29d518] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 23:45:00,490 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) [c29d518] Autorecovering 1 storage domains >2013-03-07 23:45:00,490 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) [c29d518] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 23:45:00,491 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-85) [504e04de] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:45:00,491 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-85) [504e04de] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 11:45 PM >2013-03-07 23:45:00,494 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-85) [504e04de] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 11:45 PM >2013-03-07 23:45:01,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [6c4867d5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3a4906af >2013-03-07 23:45:01,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [6c4867d5] FINISH, GetHardwareInfoVDSCommand, log id: 3a4906af >2013-03-07 23:45:01,315 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-25) [3ef806ed] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 23:45:01,319 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-25) [53c1be1f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 23:45:01,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 31f61bce >2013-03-07 23:46:06,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 23:46:06,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:46:06,298 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 23:46:06,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 31f61bce >2013-03-07 23:46:06,302 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [15ec25db] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:46:06,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [15ec25db] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 55fe1ac5 >2013-03-07 23:46:06,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [15ec25db] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 23:46:06,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [15ec25db] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 55fe1ac5 >2013-03-07 23:46:06,319 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [15ec25db] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 23:46:06,319 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [53c1be1f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 23:46:06,329 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-25) [1d879fc0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:46:06,330 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [1d879fc0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 72b4fb4e >2013-03-07 23:46:06,332 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [1d879fc0] FINISH, SetVdsStatusVDSCommand, log id: 72b4fb4e >2013-03-07 23:46:06,338 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-25) [1d879fc0] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 23:46:06,357 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-25) [6c21327c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:46:06,359 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-25) [39a060ae] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:46:06,361 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-25) [39a060ae] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 23:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [72c8fd03] Autorecovering 1 hosts >2013-03-07 23:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [72c8fd03] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [2181630a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [2181630a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [2181630a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 273951e0 >2013-03-07 23:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [2181630a] FINISH, SetVdsStatusVDSCommand, log id: 273951e0 >2013-03-07 23:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-65) [2181630a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 76a3b8b8 >2013-03-07 23:50:00,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [2181630a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2120a46f >2013-03-07 23:50:00,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [2181630a] FINISH, GetHardwareInfoVDSCommand, log id: 2120a46f >2013-03-07 23:50:00,096 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-65) [46047aef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:50:00,099 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-65) [5210b0c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:50:00,100 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-65) [5210b0c] FINISH, ActivateVdsVDSCommand, log id: 76a3b8b8 >2013-03-07 23:50:00,119 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [5210b0c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:50:00,120 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-65) [5210b0c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 23:50:00,121 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [5210b0c] Autorecovering 1 storage domains >2013-03-07 23:50:00,121 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [5210b0c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 23:50:00,122 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [27d79d50] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:50:00,122 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [27d79d50] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 11:50 PM >2013-03-07 23:50:00,125 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [27d79d50] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 11:50 PM >2013-03-07 23:50:00,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [3f1e2845] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1e98017 >2013-03-07 23:50:00,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [3f1e2845] FINISH, GetHardwareInfoVDSCommand, log id: 1e98017 >2013-03-07 23:50:00,607 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-31) [1b564bbe] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 23:50:00,610 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-31) [afdbc9e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 23:50:00,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 20edccaf >2013-03-07 23:51:03,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 23:51:03,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:51:03,232 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 23:51:03,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 20edccaf >2013-03-07 23:51:03,236 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [17309026] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:51:03,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [17309026] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 77fba6e2 >2013-03-07 23:51:03,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [17309026] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 23:51:03,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [17309026] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 77fba6e2 >2013-03-07 23:51:03,253 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [17309026] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 23:51:03,253 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [afdbc9e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 23:51:03,263 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-31) [602dece2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:51:03,265 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [602dece2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 171b5214 >2013-03-07 23:51:03,266 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [602dece2] FINISH, SetVdsStatusVDSCommand, log id: 171b5214 >2013-03-07 23:51:03,273 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-31) [602dece2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 23:51:03,293 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-31) [7ac73519] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:51:03,295 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-31) [2ab636cb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:51:03,297 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-31) [2ab636cb] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-07 23:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [7581f99e] Autorecovering 1 hosts >2013-03-07 23:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [7581f99e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [925ed3f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [925ed3f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [925ed3f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1e664e41 >2013-03-07 23:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [925ed3f] FINISH, SetVdsStatusVDSCommand, log id: 1e664e41 >2013-03-07 23:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-47) [925ed3f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2656f4f7 >2013-03-07 23:55:00,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [925ed3f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3a630c70 >2013-03-07 23:55:00,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [925ed3f] FINISH, GetHardwareInfoVDSCommand, log id: 3a630c70 >2013-03-07 23:55:00,709 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [6f469a9d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:55:00,711 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [2976f89b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:55:00,713 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-47) [2976f89b] FINISH, ActivateVdsVDSCommand, log id: 2656f4f7 >2013-03-07 23:55:00,733 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [2976f89b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-07 23:55:00,734 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [2976f89b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-07 23:55:00,735 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [2976f89b] Autorecovering 1 storage domains >2013-03-07 23:55:00,735 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [2976f89b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-07 23:55:00,735 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [5404b7e7] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:55:00,736 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [5404b7e7] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/7/13 11:55 PM >2013-03-07 23:55:00,738 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [5404b7e7] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/7/13 11:55 PM >2013-03-07 23:55:00,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [28417a3a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 38cd236b >2013-03-07 23:55:00,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [28417a3a] FINISH, GetHardwareInfoVDSCommand, log id: 38cd236b >2013-03-07 23:55:00,810 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-52) [7faf631c] Running command: InitVdsOnUpCommand internal: true. >2013-03-07 23:55:00,814 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-52) [5290c0b6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-07 23:55:00,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5f65f477 >2013-03-07 23:56:03,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-07 23:56:03,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-07 23:56:03,839 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-07 23:56:03,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 5f65f477 >2013-03-07 23:56:03,843 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [5923fcb6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-07 23:56:03,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [5923fcb6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 566892ef >2013-03-07 23:56:03,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [5923fcb6] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-07 23:56:03,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [5923fcb6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 566892ef >2013-03-07 23:56:03,859 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [5923fcb6] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-07 23:56:03,861 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [5290c0b6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-07 23:56:03,870 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-52) [52ba5efe] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:56:03,871 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [52ba5efe] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7a2f42ec >2013-03-07 23:56:03,872 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [52ba5efe] FINISH, SetVdsStatusVDSCommand, log id: 7a2f42ec >2013-03-07 23:56:03,879 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-52) [52ba5efe] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-07 23:56:03,898 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [129ede15] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:56:03,901 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [2b741d8c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-07 23:56:03,902 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-52) [2b741d8c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 00:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [1b0a3a8] Autorecovering 1 hosts >2013-03-08 00:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [1b0a3a8] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [6d658dc6] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [6d658dc6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [6d658dc6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 45f9d8e1 >2013-03-08 00:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [6d658dc6] FINISH, SetVdsStatusVDSCommand, log id: 45f9d8e1 >2013-03-08 00:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [6d658dc6] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4a21754a >2013-03-08 00:00:00,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [6d658dc6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2ee56b38 >2013-03-08 00:00:00,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [6d658dc6] FINISH, GetHardwareInfoVDSCommand, log id: 2ee56b38 >2013-03-08 00:00:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [74c2f6b8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:00:00,095 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [3c4534d8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:00:00,096 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [3c4534d8] FINISH, ActivateVdsVDSCommand, log id: 4a21754a >2013-03-08 00:00:00,115 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [3c4534d8] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:00:00,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-94) [3c4534d8] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 00:00:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [3c4534d8] Autorecovering 1 storage domains >2013-03-08 00:00:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [3c4534d8] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 00:00:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [489ff5bc] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:00:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [489ff5bc] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:00 AM >2013-03-08 00:00:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [489ff5bc] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:00 AM >2013-03-08 00:00:01,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [3742243e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 42b76625 >2013-03-08 00:00:01,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [3742243e] FINISH, GetHardwareInfoVDSCommand, log id: 42b76625 >2013-03-08 00:00:01,154 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-91) [75ffeff3] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 00:00:01,158 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-91) [21ae1b7d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 00:00:01,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 149e50d9 >2013-03-08 00:01:06,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 00:01:06,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:01:06,197 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 00:01:06,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 149e50d9 >2013-03-08 00:01:06,200 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [34df751b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:01:06,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [34df751b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 659af9b2 >2013-03-08 00:01:06,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [34df751b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 00:01:06,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [34df751b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 659af9b2 >2013-03-08 00:01:06,217 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [34df751b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 00:01:06,217 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [21ae1b7d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 00:01:06,319 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-91) [38fc8326] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:01:06,320 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [38fc8326] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1a23832 >2013-03-08 00:01:06,322 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [38fc8326] FINISH, SetVdsStatusVDSCommand, log id: 1a23832 >2013-03-08 00:01:06,331 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-91) [38fc8326] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 00:01:06,350 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-91) [61a22c21] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:01:06,352 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-91) [1899bf25] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:01:06,353 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-91) [1899bf25] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 00:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [65873a9c] Autorecovering 1 hosts >2013-03-08 00:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [65873a9c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [14c3423a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [14c3423a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [14c3423a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 795bff14 >2013-03-08 00:05:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [14c3423a] FINISH, SetVdsStatusVDSCommand, log id: 795bff14 >2013-03-08 00:05:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-68) [14c3423a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2b9dc3d0 >2013-03-08 00:05:00,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [14c3423a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 27d24887 >2013-03-08 00:05:00,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [14c3423a] FINISH, GetHardwareInfoVDSCommand, log id: 27d24887 >2013-03-08 00:05:00,490 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-68) [3d5f837a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:05:00,493 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-68) [61f11121] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:05:00,494 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-68) [61f11121] FINISH, ActivateVdsVDSCommand, log id: 2b9dc3d0 >2013-03-08 00:05:00,517 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [61f11121] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:05:00,518 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-68) [61f11121] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 00:05:00,519 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [61f11121] Autorecovering 1 storage domains >2013-03-08 00:05:00,519 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [61f11121] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 00:05:00,520 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [3ae408a5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:05:00,520 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [3ae408a5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:05 AM >2013-03-08 00:05:00,522 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [3ae408a5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:05 AM >2013-03-08 00:05:00,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [5487f815] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 8290e49 >2013-03-08 00:05:00,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [5487f815] FINISH, GetHardwareInfoVDSCommand, log id: 8290e49 >2013-03-08 00:05:00,601 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-8) [129a42e6] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 00:05:00,604 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-8) [cbbc99e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 00:05:00,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1945826c >2013-03-08 00:06:04,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 00:06:04,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:06:04,610 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 00:06:04,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 1945826c >2013-03-08 00:06:04,614 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [187bdca3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:06:04,635 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [187bdca3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 72ea58ce >2013-03-08 00:06:04,635 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [187bdca3] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 00:06:04,635 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [187bdca3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 72ea58ce >2013-03-08 00:06:04,637 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [187bdca3] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 00:06:04,637 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [cbbc99e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 00:06:04,647 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-8) [7be7ca90] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:06:04,649 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [7be7ca90] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 229d27fa >2013-03-08 00:06:04,650 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [7be7ca90] FINISH, SetVdsStatusVDSCommand, log id: 229d27fa >2013-03-08 00:06:04,656 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-8) [7be7ca90] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 00:06:04,677 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [63c8da22] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:06:04,680 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [1ba6ea61] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:06:04,681 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-8) [1ba6ea61] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 00:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [6c7f2de4] Autorecovering 1 hosts >2013-03-08 00:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [6c7f2de4] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-56) [294999ba] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-56) [294999ba] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [294999ba] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 674ade36 >2013-03-08 00:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [294999ba] FINISH, SetVdsStatusVDSCommand, log id: 674ade36 >2013-03-08 00:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-56) [294999ba] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 57605ce >2013-03-08 00:10:00,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [294999ba] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 78401c60 >2013-03-08 00:10:00,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [294999ba] FINISH, GetHardwareInfoVDSCommand, log id: 78401c60 >2013-03-08 00:10:00,096 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-56) [b222d96] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:10:00,099 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-56) [37499c02] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:10:00,100 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-56) [37499c02] FINISH, ActivateVdsVDSCommand, log id: 57605ce >2013-03-08 00:10:00,119 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-56) [37499c02] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:10:00,120 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-56) [37499c02] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 00:10:00,121 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [37499c02] Autorecovering 1 storage domains >2013-03-08 00:10:00,121 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [37499c02] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 00:10:00,122 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-56) [2b30cfaa] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:10:00,122 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-56) [2b30cfaa] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:10 AM >2013-03-08 00:10:00,125 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-56) [2b30cfaa] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:10 AM >2013-03-08 00:10:01,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [74ce696f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 53590e4d >2013-03-08 00:10:01,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [74ce696f] FINISH, GetHardwareInfoVDSCommand, log id: 53590e4d >2013-03-08 00:10:01,936 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-26) [3badb6d3] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 00:10:01,940 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-26) [6ff8b5f8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 00:10:01,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2e025132 >2013-03-08 00:11:06,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 00:11:06,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:11:06,951 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 00:11:06,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 2e025132 >2013-03-08 00:11:06,955 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [2179fd63] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:11:06,969 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [2179fd63] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 72c9446 >2013-03-08 00:11:06,969 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [2179fd63] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 00:11:06,969 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [2179fd63] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 72c9446 >2013-03-08 00:11:06,976 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [6ff8b5f8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 00:11:06,984 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-26) [375fcd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:11:06,985 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [375fcd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 30d14652 >2013-03-08 00:11:06,987 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [375fcd] FINISH, SetVdsStatusVDSCommand, log id: 30d14652 >2013-03-08 00:11:06,992 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [375fcd] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 00:11:07,012 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [7078119a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:11:07,015 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [1efa21cb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:11:07,016 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-26) [1efa21cb] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 00:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [7b678968] Autorecovering 1 hosts >2013-03-08 00:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [7b678968] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [6935a274] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [6935a274] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [6935a274] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2a8da6f8 >2013-03-08 00:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [6935a274] FINISH, SetVdsStatusVDSCommand, log id: 2a8da6f8 >2013-03-08 00:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-33) [6935a274] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6f4af56b >2013-03-08 00:15:00,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [6935a274] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4892a9b4 >2013-03-08 00:15:00,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [6935a274] FINISH, GetHardwareInfoVDSCommand, log id: 4892a9b4 >2013-03-08 00:15:00,489 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-33) [2cd23bca] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:15:00,491 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-33) [6fa6e260] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:15:00,493 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-33) [6fa6e260] FINISH, ActivateVdsVDSCommand, log id: 6f4af56b >2013-03-08 00:15:00,512 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [6fa6e260] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:15:00,516 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [6fa6e260] Autorecovering 1 storage domains >2013-03-08 00:15:00,516 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [6fa6e260] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 00:15:00,517 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [59cf7b26] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:15:00,517 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [59cf7b26] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:15 AM >2013-03-08 00:15:00,520 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [59cf7b26] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:15 AM >2013-03-08 00:15:01,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [3099d800] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1c0d5e7c >2013-03-08 00:15:01,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [3099d800] FINISH, GetHardwareInfoVDSCommand, log id: 1c0d5e7c >2013-03-08 00:15:01,260 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-64) [4f61762a] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 00:15:01,263 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-64) [1ba42a8f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 00:15:01,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6c091e22 >2013-03-08 00:16:05,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 00:16:05,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:16:05,567 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 00:16:05,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 6c091e22 >2013-03-08 00:16:05,572 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [15c163] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:16:05,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [15c163] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3de526ee >2013-03-08 00:16:05,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [15c163] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 00:16:05,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [15c163] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3de526ee >2013-03-08 00:16:05,592 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [15c163] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 00:16:05,592 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [1ba42a8f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 00:16:05,606 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-64) [c7bf42e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:16:05,607 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [c7bf42e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2235ca3e >2013-03-08 00:16:05,609 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [c7bf42e] FINISH, SetVdsStatusVDSCommand, log id: 2235ca3e >2013-03-08 00:16:05,615 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-64) [c7bf42e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 00:16:05,639 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-64) [3e2e9698] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:16:05,642 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-64) [41ee8073] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:16:05,643 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-64) [41ee8073] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 00:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [75937e8e] Autorecovering 1 hosts >2013-03-08 00:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [75937e8e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [34816278] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [34816278] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [34816278] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: a6b4ca0 >2013-03-08 00:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [34816278] FINISH, SetVdsStatusVDSCommand, log id: a6b4ca0 >2013-03-08 00:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-50) [34816278] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 32dd7ed4 >2013-03-08 00:20:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [34816278] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3411d6db >2013-03-08 00:20:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [34816278] FINISH, GetHardwareInfoVDSCommand, log id: 3411d6db >2013-03-08 00:20:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-50) [1439753] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:20:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-50) [46643506] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:20:00,093 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-50) [46643506] FINISH, ActivateVdsVDSCommand, log id: 32dd7ed4 >2013-03-08 00:20:00,113 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [46643506] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:20:00,114 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-50) [46643506] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 00:20:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [46643506] Autorecovering 1 storage domains >2013-03-08 00:20:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [46643506] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 00:20:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [30d2aa24] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:20:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [30d2aa24] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:20 AM >2013-03-08 00:20:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [30d2aa24] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:20 AM >2013-03-08 00:20:02,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [3a994fb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 733199f2 >2013-03-08 00:20:02,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [3a994fb] FINISH, GetHardwareInfoVDSCommand, log id: 733199f2 >2013-03-08 00:20:02,896 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-43) [6aadb59e] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 00:20:02,899 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-43) [243027b2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 00:20:02,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3e23cc43 >2013-03-08 00:21:07,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 00:21:07,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:21:07,905 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 00:21:07,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 3e23cc43 >2013-03-08 00:21:07,909 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [43b0b677] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:21:07,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [43b0b677] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4b819dd4 >2013-03-08 00:21:07,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [43b0b677] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 00:21:07,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [43b0b677] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4b819dd4 >2013-03-08 00:21:07,926 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [43b0b677] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 00:21:07,927 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [243027b2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 00:21:07,937 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-43) [43967176] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:21:07,938 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [43967176] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2811f9b4 >2013-03-08 00:21:07,940 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [43967176] FINISH, SetVdsStatusVDSCommand, log id: 2811f9b4 >2013-03-08 00:21:07,947 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [43967176] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 00:21:07,971 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [7c54b71b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:21:07,974 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [434b4905] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:21:07,975 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-43) [434b4905] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 00:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [225c2169] Autorecovering 1 hosts >2013-03-08 00:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [225c2169] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [237b7d14] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [237b7d14] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [237b7d14] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1d5a7481 >2013-03-08 00:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [237b7d14] FINISH, SetVdsStatusVDSCommand, log id: 1d5a7481 >2013-03-08 00:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-57) [237b7d14] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 30f76f4e >2013-03-08 00:25:00,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [237b7d14] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 52717cc8 >2013-03-08 00:25:00,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [237b7d14] FINISH, GetHardwareInfoVDSCommand, log id: 52717cc8 >2013-03-08 00:25:00,100 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-57) [eba077b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:25:00,103 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-57) [437f7c53] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:25:00,104 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-57) [437f7c53] FINISH, ActivateVdsVDSCommand, log id: 30f76f4e >2013-03-08 00:25:00,123 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [437f7c53] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:25:00,124 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-57) [437f7c53] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 00:25:00,125 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [437f7c53] Autorecovering 1 storage domains >2013-03-08 00:25:00,125 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [437f7c53] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 00:25:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [69e2dae5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:25:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [69e2dae5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:25 AM >2013-03-08 00:25:00,129 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [69e2dae5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:25 AM >2013-03-08 00:25:02,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [6b31e9fe] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 521580a0 >2013-03-08 00:25:02,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [6b31e9fe] FINISH, GetHardwareInfoVDSCommand, log id: 521580a0 >2013-03-08 00:25:02,224 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-70) [13148852] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 00:25:02,228 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-70) [571afc59] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 00:25:02,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 635fd0eb >2013-03-08 00:26:06,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 00:26:06,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:26:06,551 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 00:26:06,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 635fd0eb >2013-03-08 00:26:06,554 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [3fed12d9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:26:06,574 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [3fed12d9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3a89bca >2013-03-08 00:26:06,574 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [3fed12d9] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 00:26:06,574 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [3fed12d9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3a89bca >2013-03-08 00:26:06,576 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [3fed12d9] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 00:26:06,577 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [571afc59] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 00:26:06,586 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-70) [5dedba2f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:26:06,587 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [5dedba2f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6a28e11b >2013-03-08 00:26:06,589 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [5dedba2f] FINISH, SetVdsStatusVDSCommand, log id: 6a28e11b >2013-03-08 00:26:06,596 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-70) [5dedba2f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 00:26:06,616 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-70) [5c5adc0b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:26:06,619 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-70) [693404a1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:26:06,620 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-70) [693404a1] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 00:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [363a8986] Autorecovering 1 hosts >2013-03-08 00:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [363a8986] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [404bcbea] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [404bcbea] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [404bcbea] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1fab992 >2013-03-08 00:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [404bcbea] FINISH, SetVdsStatusVDSCommand, log id: 1fab992 >2013-03-08 00:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-41) [404bcbea] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3b5e3f08 >2013-03-08 00:30:00,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [404bcbea] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2cc514d6 >2013-03-08 00:30:00,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [404bcbea] FINISH, GetHardwareInfoVDSCommand, log id: 2cc514d6 >2013-03-08 00:30:00,678 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-41) [58057f8f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:30:00,681 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-41) [5911193f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:30:00,682 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-41) [5911193f] FINISH, ActivateVdsVDSCommand, log id: 3b5e3f08 >2013-03-08 00:30:00,701 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [5911193f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:30:00,702 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-41) [5911193f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 00:30:00,703 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [5911193f] Autorecovering 1 storage domains >2013-03-08 00:30:00,703 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [5911193f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 00:30:00,704 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [45fa64d6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:30:00,704 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [45fa64d6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:30 AM >2013-03-08 00:30:00,707 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [45fa64d6] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:30 AM >2013-03-08 00:30:00,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [679c2827] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: f3bbe66 >2013-03-08 00:30:00,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [679c2827] FINISH, GetHardwareInfoVDSCommand, log id: f3bbe66 >2013-03-08 00:30:00,867 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-2) [7e63af08] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 00:30:00,870 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-2) [e049dc5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 00:30:00,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6a4adb21 >2013-03-08 00:30:33,713 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-52) [2b741d8c] Start refreshing all users data >2013-03-08 00:30:34,620 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-75) [ec4766d] Attempting to update VMs/Templates Ovf. >2013-03-08 00:30:34,621 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-75) [ec4766d] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 00:30:34,622 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-75) [ec4766d] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 00:30:34,622 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-75) [ec4766d] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 00:30:34,623 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-75) [ec4766d] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 00:30:34,623 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-75) [ec4766d] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 00:30:34,624 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-75) [ec4766d] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 00:31:04,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 00:31:04,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:31:04,865 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 00:31:04,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 6a4adb21 >2013-03-08 00:31:04,869 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [732316e2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:31:04,884 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [732316e2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5cc11c6a >2013-03-08 00:31:04,884 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [732316e2] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 00:31:04,884 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [732316e2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5cc11c6a >2013-03-08 00:31:04,886 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [732316e2] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 00:31:04,886 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [e049dc5] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 00:31:04,896 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-2) [66ad719b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:31:04,898 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [66ad719b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2713734a >2013-03-08 00:31:04,899 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [66ad719b] FINISH, SetVdsStatusVDSCommand, log id: 2713734a >2013-03-08 00:31:04,906 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [66ad719b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 00:31:04,925 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [3a743475] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:31:04,927 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [6d273a42] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:31:04,928 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-2) [6d273a42] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 00:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [6c49d26] Autorecovering 1 hosts >2013-03-08 00:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [6c49d26] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-4) [7d5985d2] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-4) [7d5985d2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [7d5985d2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2fcc552 >2013-03-08 00:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [7d5985d2] FINISH, SetVdsStatusVDSCommand, log id: 2fcc552 >2013-03-08 00:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-4) [7d5985d2] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1cbe7125 >2013-03-08 00:35:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [7d5985d2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 15ec4e35 >2013-03-08 00:35:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [7d5985d2] FINISH, GetHardwareInfoVDSCommand, log id: 15ec4e35 >2013-03-08 00:35:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-4) [7151b754] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:35:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-4) [2575812] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:35:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-4) [2575812] FINISH, ActivateVdsVDSCommand, log id: 1cbe7125 >2013-03-08 00:35:00,111 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-4) [2575812] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:35:00,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-4) [2575812] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 00:35:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [2575812] Autorecovering 1 storage domains >2013-03-08 00:35:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [2575812] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 00:35:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-4) [2658832] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:35:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-4) [2658832] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:35 AM >2013-03-08 00:35:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-4) [2658832] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:35 AM >2013-03-08 00:35:02,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [2510b209] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 46db93b6 >2013-03-08 00:35:02,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [2510b209] FINISH, GetHardwareInfoVDSCommand, log id: 46db93b6 >2013-03-08 00:35:02,176 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-32) [4ab08a1c] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 00:35:02,179 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-32) [46a8f1bf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 00:35:02,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5f16369e >2013-03-08 00:36:07,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 00:36:07,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:36:07,167 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 00:36:07,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 5f16369e >2013-03-08 00:36:07,171 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [3f0417e4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:36:07,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [3f0417e4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 16418628 >2013-03-08 00:36:07,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [3f0417e4] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 00:36:07,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [3f0417e4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 16418628 >2013-03-08 00:36:07,189 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [3f0417e4] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 00:36:07,189 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [46a8f1bf] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 00:36:07,200 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-32) [31d0c7de] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:36:07,202 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [31d0c7de] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 79e27106 >2013-03-08 00:36:07,203 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [31d0c7de] FINISH, SetVdsStatusVDSCommand, log id: 79e27106 >2013-03-08 00:36:07,209 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-32) [31d0c7de] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 00:36:07,230 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-32) [915ca42] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:36:07,232 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-32) [75293efe] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:36:07,234 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-32) [75293efe] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 00:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [4f9bf40f] Autorecovering 1 hosts >2013-03-08 00:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [4f9bf40f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-3) [1a4eafeb] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-3) [1a4eafeb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-3) [1a4eafeb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 521214b4 >2013-03-08 00:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-3) [1a4eafeb] FINISH, SetVdsStatusVDSCommand, log id: 521214b4 >2013-03-08 00:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-3) [1a4eafeb] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: ff0f723 >2013-03-08 00:40:00,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-3) [1a4eafeb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 702dfc69 >2013-03-08 00:40:00,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-3) [1a4eafeb] FINISH, GetHardwareInfoVDSCommand, log id: 702dfc69 >2013-03-08 00:40:00,535 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-3) [1be9e311] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:40:00,538 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-3) [540cce35] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:40:00,539 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-3) [540cce35] FINISH, ActivateVdsVDSCommand, log id: ff0f723 >2013-03-08 00:40:00,559 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-3) [540cce35] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:40:00,560 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) [540cce35] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 00:40:00,561 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [540cce35] Autorecovering 1 storage domains >2013-03-08 00:40:00,561 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [540cce35] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 00:40:00,562 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-3) [d093265] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:40:00,562 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-3) [d093265] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:40 AM >2013-03-08 00:40:00,565 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-3) [d093265] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:40 AM >2013-03-08 00:40:01,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [66bbed93] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1075f8e7 >2013-03-08 00:40:01,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [66bbed93] FINISH, GetHardwareInfoVDSCommand, log id: 1075f8e7 >2013-03-08 00:40:01,507 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-15) [29a738c2] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 00:40:01,511 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-15) [422a0677] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 00:40:01,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 478513f4 >2013-03-08 00:41:05,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 00:41:05,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:41:05,497 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 00:41:05,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 478513f4 >2013-03-08 00:41:05,501 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [596b06f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:41:05,515 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [596b06f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 100baec3 >2013-03-08 00:41:05,515 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [596b06f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 00:41:05,515 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [596b06f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 100baec3 >2013-03-08 00:41:05,517 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [596b06f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 00:41:05,517 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [422a0677] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 00:41:05,528 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-15) [79703432] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:41:05,529 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [79703432] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5aa1914a >2013-03-08 00:41:05,530 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [79703432] FINISH, SetVdsStatusVDSCommand, log id: 5aa1914a >2013-03-08 00:41:05,536 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-15) [79703432] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 00:41:05,556 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-15) [22ff6f75] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:41:05,558 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-15) [804dd00] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:41:05,559 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-15) [804dd00] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 00:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [66e3f062] Autorecovering 1 hosts >2013-03-08 00:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [66e3f062] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-82) [777e9f42] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-82) [777e9f42] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [777e9f42] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 72f43e41 >2013-03-08 00:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [777e9f42] FINISH, SetVdsStatusVDSCommand, log id: 72f43e41 >2013-03-08 00:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-82) [777e9f42] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 750e606d >2013-03-08 00:45:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [777e9f42] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3f28b494 >2013-03-08 00:45:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [777e9f42] FINISH, GetHardwareInfoVDSCommand, log id: 3f28b494 >2013-03-08 00:45:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-82) [2f36f62] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:45:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-82) [777c19a1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:45:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-82) [777c19a1] FINISH, ActivateVdsVDSCommand, log id: 750e606d >2013-03-08 00:45:00,111 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-82) [777c19a1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:45:00,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-82) [777c19a1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 00:45:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [777c19a1] Autorecovering 1 storage domains >2013-03-08 00:45:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [777c19a1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 00:45:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-82) [4a7d529f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:45:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-82) [4a7d529f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:45 AM >2013-03-08 00:45:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-82) [4a7d529f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:45 AM >2013-03-08 00:45:02,751 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3cc88b69 >2013-03-08 00:45:02,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) FINISH, GetHardwareInfoVDSCommand, log id: 3cc88b69 >2013-03-08 00:45:02,810 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-53) [24d15049] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 00:45:02,813 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-53) [2b29cb14] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 00:45:02,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 566b0baf >2013-03-08 00:46:06,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 00:46:06,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:46:06,800 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 00:46:06,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 566b0baf >2013-03-08 00:46:06,804 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [1736c2ab] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:46:06,820 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [1736c2ab] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3e33f2cd >2013-03-08 00:46:06,820 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [1736c2ab] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 00:46:06,820 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [1736c2ab] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3e33f2cd >2013-03-08 00:46:06,822 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [1736c2ab] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 00:46:06,822 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [2b29cb14] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 00:46:06,832 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-53) [1fae4d02] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:46:06,833 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [1fae4d02] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7cd0a644 >2013-03-08 00:46:06,835 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [1fae4d02] FINISH, SetVdsStatusVDSCommand, log id: 7cd0a644 >2013-03-08 00:46:06,840 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-53) [1fae4d02] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 00:46:06,865 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-53) [7728d220] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:46:06,867 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-53) [34358fb3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:46:06,869 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-53) [34358fb3] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 00:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) Autorecovering 1 hosts >2013-03-08 00:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-95) [7d790698] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-95) [7d790698] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [7d790698] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 81c4ec >2013-03-08 00:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [7d790698] FINISH, SetVdsStatusVDSCommand, log id: 81c4ec >2013-03-08 00:50:00,018 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-95) [7d790698] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7a8d0e94 >2013-03-08 00:50:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [7d790698] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 54cc3f3a >2013-03-08 00:50:00,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [7d790698] FINISH, GetHardwareInfoVDSCommand, log id: 54cc3f3a >2013-03-08 00:50:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-95) [606e55f2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:50:00,208 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-95) [4eef2f75] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:50:00,210 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-95) [4eef2f75] FINISH, ActivateVdsVDSCommand, log id: 7a8d0e94 >2013-03-08 00:50:00,229 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-95) [4eef2f75] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:50:00,230 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-95) [4eef2f75] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 00:50:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [4eef2f75] Autorecovering 1 storage domains >2013-03-08 00:50:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [4eef2f75] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 00:50:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-95) [35d63329] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:50:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-95) [35d63329] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:50 AM >2013-03-08 00:50:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-95) [35d63329] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:50 AM >2013-03-08 00:50:01,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [68d98cc8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 70555143 >2013-03-08 00:50:01,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [68d98cc8] FINISH, GetHardwareInfoVDSCommand, log id: 70555143 >2013-03-08 00:50:01,110 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-100) [2f58be67] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 00:50:01,114 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-100) [5755b47b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 00:50:01,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 21357f9d >2013-03-08 00:51:07,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 00:51:07,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:51:07,162 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 00:51:07,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 21357f9d >2013-03-08 00:51:07,165 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [50c04de4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:51:07,180 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [50c04de4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 30555264 >2013-03-08 00:51:07,180 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [50c04de4] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 00:51:07,180 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [50c04de4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 30555264 >2013-03-08 00:51:07,182 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [50c04de4] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 00:51:07,183 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [5755b47b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 00:51:07,192 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-100) [7c0b7c61] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:51:07,193 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [7c0b7c61] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 786f9b99 >2013-03-08 00:51:07,195 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [7c0b7c61] FINISH, SetVdsStatusVDSCommand, log id: 786f9b99 >2013-03-08 00:51:07,200 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-100) [7c0b7c61] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 00:51:07,219 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-100) [38e087d3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:51:07,222 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-100) [1e8bb7c7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:51:07,223 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-100) [1e8bb7c7] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 00:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [11a87ed8] Autorecovering 1 hosts >2013-03-08 00:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [11a87ed8] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [14984f63] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [14984f63] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [14984f63] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7b389c8e >2013-03-08 00:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [14984f63] FINISH, SetVdsStatusVDSCommand, log id: 7b389c8e >2013-03-08 00:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [14984f63] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1f511d36 >2013-03-08 00:55:00,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [14984f63] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: c9aab9d >2013-03-08 00:55:00,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [14984f63] FINISH, GetHardwareInfoVDSCommand, log id: c9aab9d >2013-03-08 00:55:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-73) [d13efcf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:55:00,206 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-73) [25bd6283] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:55:00,208 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [25bd6283] FINISH, ActivateVdsVDSCommand, log id: 1f511d36 >2013-03-08 00:55:00,227 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [25bd6283] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 00:55:00,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-73) [25bd6283] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 00:55:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [25bd6283] Autorecovering 1 storage domains >2013-03-08 00:55:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [25bd6283] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 00:55:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [638bb2a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:55:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [638bb2a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:55 AM >2013-03-08 00:55:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [638bb2a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:55 AM >2013-03-08 00:55:01,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [7e412d47] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 313aca42 >2013-03-08 00:55:01,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [7e412d47] FINISH, GetHardwareInfoVDSCommand, log id: 313aca42 >2013-03-08 00:55:01,471 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-88) [4868adff] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 00:55:01,474 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-88) [23469195] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 00:55:01,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 73683f1c >2013-03-08 00:56:06,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 00:56:06,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 00:56:06,472 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 00:56:06,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 73683f1c >2013-03-08 00:56:06,475 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [8839d2c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 00:56:06,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [8839d2c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: eaff65a >2013-03-08 00:56:06,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [8839d2c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 00:56:06,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [8839d2c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: eaff65a >2013-03-08 00:56:06,492 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [8839d2c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 00:56:06,493 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [23469195] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 00:56:06,504 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-88) [3c945011] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:56:06,505 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [3c945011] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 17595e37 >2013-03-08 00:56:06,506 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [3c945011] FINISH, SetVdsStatusVDSCommand, log id: 17595e37 >2013-03-08 00:56:06,513 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-88) [3c945011] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 00:56:06,533 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-88) [10c9372f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:56:06,535 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-88) [6930a36d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 00:56:06,537 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-88) [6930a36d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 01:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [364f59a5] Autorecovering 1 hosts >2013-03-08 01:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [364f59a5] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [65f9727] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [65f9727] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [65f9727] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: f34781f >2013-03-08 01:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [65f9727] FINISH, SetVdsStatusVDSCommand, log id: f34781f >2013-03-08 01:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-58) [65f9727] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 533cd112 >2013-03-08 01:00:00,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [65f9727] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 532b897e >2013-03-08 01:00:00,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [65f9727] FINISH, GetHardwareInfoVDSCommand, log id: 532b897e >2013-03-08 01:00:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-58) [4ef87729] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:00:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-58) [192c85c8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:00:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-58) [192c85c8] FINISH, ActivateVdsVDSCommand, log id: 533cd112 >2013-03-08 01:00:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [192c85c8] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:00:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-58) [192c85c8] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 01:00:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [192c85c8] Autorecovering 1 storage domains >2013-03-08 01:00:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [192c85c8] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 01:00:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [4ca19890] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:00:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [4ca19890] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:00 AM >2013-03-08 01:00:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [4ca19890] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:00 AM >2013-03-08 01:00:00,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [68139580] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2085f4eb >2013-03-08 01:00:00,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [68139580] FINISH, GetHardwareInfoVDSCommand, log id: 2085f4eb >2013-03-08 01:00:00,780 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-28) [3b4d2cca] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 01:00:00,783 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-28) [4459728e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 01:00:00,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4b0970e7 >2013-03-08 01:01:05,787 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 01:01:05,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:01:05,788 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 01:01:05,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 4b0970e7 >2013-03-08 01:01:05,792 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [28aceb3d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:01:05,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [28aceb3d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 68b49114 >2013-03-08 01:01:05,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [28aceb3d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 01:01:05,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [28aceb3d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 68b49114 >2013-03-08 01:01:05,809 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [28aceb3d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 01:01:05,809 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [4459728e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 01:01:05,821 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-28) [75b9eae4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:01:05,822 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [75b9eae4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7f03db34 >2013-03-08 01:01:05,823 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [75b9eae4] FINISH, SetVdsStatusVDSCommand, log id: 7f03db34 >2013-03-08 01:01:05,828 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-28) [75b9eae4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 01:01:05,850 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-28) [3916f2ee] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:01:05,852 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-28) [34620c93] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:01:05,854 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-28) [34620c93] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 01:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [ec4766d] Autorecovering 1 hosts >2013-03-08 01:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [ec4766d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [3c99b554] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [3c99b554] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [3c99b554] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3bbbbb9e >2013-03-08 01:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [3c99b554] FINISH, SetVdsStatusVDSCommand, log id: 3bbbbb9e >2013-03-08 01:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-75) [3c99b554] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1d8a037d >2013-03-08 01:05:00,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [3c99b554] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 28b01f86 >2013-03-08 01:05:00,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [3c99b554] FINISH, GetHardwareInfoVDSCommand, log id: 28b01f86 >2013-03-08 01:05:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [7cd1ec41] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:05:00,263 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [2d257ccd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:05:00,264 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-75) [2d257ccd] FINISH, ActivateVdsVDSCommand, log id: 1d8a037d >2013-03-08 01:05:00,283 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [2d257ccd] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:05:00,284 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-75) [2d257ccd] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 01:05:00,286 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [2d257ccd] Autorecovering 1 storage domains >2013-03-08 01:05:00,286 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [2d257ccd] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 01:05:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [3a39221f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:05:00,287 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [3a39221f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:05 AM >2013-03-08 01:05:00,289 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [3a39221f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:05 AM >2013-03-08 01:05:03,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-40) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7a790841 >2013-03-08 01:05:03,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-40) FINISH, GetHardwareInfoVDSCommand, log id: 7a790841 >2013-03-08 01:05:03,106 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-40) [16bda4c3] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 01:05:03,109 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-40) [3e3abe2a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 01:05:03,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 75c59483 >2013-03-08 01:06:07,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 01:06:07,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:06:07,116 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 01:06:07,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 75c59483 >2013-03-08 01:06:07,120 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [3a2a3cc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:06:07,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [3a2a3cc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 514a3a0f >2013-03-08 01:06:07,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [3a2a3cc] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 01:06:07,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [3a2a3cc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 514a3a0f >2013-03-08 01:06:07,141 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [3a2a3cc] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 01:06:07,141 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [3e3abe2a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 01:06:07,164 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-40) [14218189] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:06:07,166 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-40) [14218189] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 68a47a2c >2013-03-08 01:06:07,168 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-40) [14218189] FINISH, SetVdsStatusVDSCommand, log id: 68a47a2c >2013-03-08 01:06:07,178 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-40) [14218189] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 01:06:07,213 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-40) [34a2b07c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:06:07,216 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-40) [7037220e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:06:07,217 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-40) [7037220e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 01:10:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [77d00afb] Autorecovering 1 hosts >2013-03-08 01:10:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [77d00afb] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:10:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [51f667d0] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:10:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [51f667d0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [51f667d0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 35ebc095 >2013-03-08 01:10:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [51f667d0] FINISH, SetVdsStatusVDSCommand, log id: 35ebc095 >2013-03-08 01:10:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [51f667d0] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6da345fc >2013-03-08 01:10:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [51f667d0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1044d020 >2013-03-08 01:10:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [51f667d0] FINISH, GetHardwareInfoVDSCommand, log id: 1044d020 >2013-03-08 01:10:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-14) [403c89f8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:10:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-14) [580d38b2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:10:00,205 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [580d38b2] FINISH, ActivateVdsVDSCommand, log id: 6da345fc >2013-03-08 01:10:00,227 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [580d38b2] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:10:00,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-14) [580d38b2] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 01:10:00,232 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [580d38b2] Autorecovering 1 storage domains >2013-03-08 01:10:00,232 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [580d38b2] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 01:10:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [3858662b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:10:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [3858662b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:10 AM >2013-03-08 01:10:00,252 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [3858662b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:10 AM >2013-03-08 01:10:01,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [68e6cce3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 57caca76 >2013-03-08 01:10:01,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [68e6cce3] FINISH, GetHardwareInfoVDSCommand, log id: 57caca76 >2013-03-08 01:10:01,491 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-16) [44a62f5] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 01:10:01,495 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-16) [3b8a10cc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 01:10:01,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4ded3997 >2013-03-08 01:11:05,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 01:11:05,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:11:05,516 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 01:11:05,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 4ded3997 >2013-03-08 01:11:05,520 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [11bf86df] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:11:05,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [11bf86df] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 64311e1d >2013-03-08 01:11:05,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [11bf86df] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 01:11:05,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [11bf86df] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 64311e1d >2013-03-08 01:11:05,536 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [11bf86df] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 01:11:05,537 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [3b8a10cc] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 01:11:05,546 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-16) [4062ecbd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:11:05,548 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [4062ecbd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3c6b4979 >2013-03-08 01:11:05,549 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [4062ecbd] FINISH, SetVdsStatusVDSCommand, log id: 3c6b4979 >2013-03-08 01:11:05,554 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-16) [4062ecbd] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 01:11:05,580 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-16) [211ca0c7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:11:05,582 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-16) [3b606a7f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:11:05,584 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-16) [3b606a7f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 01:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [1ba6ea61] Autorecovering 1 hosts >2013-03-08 01:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [1ba6ea61] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [41752caf] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [41752caf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [41752caf] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2ee3d0ea >2013-03-08 01:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [41752caf] FINISH, SetVdsStatusVDSCommand, log id: 2ee3d0ea >2013-03-08 01:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [41752caf] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1549a35d >2013-03-08 01:15:00,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [41752caf] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7e3b21d7 >2013-03-08 01:15:00,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [41752caf] FINISH, GetHardwareInfoVDSCommand, log id: 7e3b21d7 >2013-03-08 01:15:00,261 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [3c5077e3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:15:00,263 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [dccd6a2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:15:00,264 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [dccd6a2] FINISH, ActivateVdsVDSCommand, log id: 1549a35d >2013-03-08 01:15:00,290 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [dccd6a2] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:15:00,291 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-8) [dccd6a2] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 01:15:00,292 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [dccd6a2] Autorecovering 1 storage domains >2013-03-08 01:15:00,292 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [dccd6a2] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 01:15:00,293 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [1bd8a9ae] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:15:00,293 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [1bd8a9ae] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:15 AM >2013-03-08 01:15:00,296 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [1bd8a9ae] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:15 AM >2013-03-08 01:15:02,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [576edfad] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2ad208a6 >2013-03-08 01:15:02,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [576edfad] FINISH, GetHardwareInfoVDSCommand, log id: 2ad208a6 >2013-03-08 01:15:02,847 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-69) [38a3c9b3] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 01:15:02,851 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-69) [3945a721] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 01:15:02,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 417df82f >2013-03-08 01:16:06,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 01:16:06,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:16:06,826 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 01:16:06,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 417df82f >2013-03-08 01:16:06,830 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [62c0ef0a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:16:06,844 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [62c0ef0a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 15f84535 >2013-03-08 01:16:06,844 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [62c0ef0a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 01:16:06,844 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [62c0ef0a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 15f84535 >2013-03-08 01:16:06,849 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [3945a721] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 01:16:06,859 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-69) [e10c2ba] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:16:06,861 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [e10c2ba] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: e385242 >2013-03-08 01:16:06,862 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [e10c2ba] FINISH, SetVdsStatusVDSCommand, log id: e385242 >2013-03-08 01:16:06,868 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-69) [e10c2ba] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 01:16:06,887 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-69) [163f20be] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:16:06,890 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-69) [39799c8b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:16:06,891 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-69) [39799c8b] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 01:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [2287daa3] Autorecovering 1 hosts >2013-03-08 01:20:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [2287daa3] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-12) [92a160d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-12) [92a160d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [92a160d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: f982e23 >2013-03-08 01:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [92a160d] FINISH, SetVdsStatusVDSCommand, log id: f982e23 >2013-03-08 01:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-12) [92a160d] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2867f951 >2013-03-08 01:20:00,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [92a160d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7befb397 >2013-03-08 01:20:00,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [92a160d] FINISH, GetHardwareInfoVDSCommand, log id: 7befb397 >2013-03-08 01:20:00,406 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-12) [43ad9ab5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:20:00,409 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-12) [772d2b71] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:20:00,410 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-12) [772d2b71] FINISH, ActivateVdsVDSCommand, log id: 2867f951 >2013-03-08 01:20:00,430 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-12) [772d2b71] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:20:00,431 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-12) [772d2b71] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 01:20:00,432 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [772d2b71] Autorecovering 1 storage domains >2013-03-08 01:20:00,432 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [772d2b71] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 01:20:00,433 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-12) [4febce43] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:20:00,433 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-12) [4febce43] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:20 AM >2013-03-08 01:20:00,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-12) [4febce43] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:20 AM >2013-03-08 01:20:01,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [12f5a35f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2777aac1 >2013-03-08 01:20:01,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [12f5a35f] FINISH, GetHardwareInfoVDSCommand, log id: 2777aac1 >2013-03-08 01:20:01,142 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-10) [1e1afff2] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 01:20:01,145 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-10) [16409e5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 01:20:01,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6042c458 >2013-03-08 01:21:05,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 01:21:05,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:21:05,150 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 01:21:05,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 6042c458 >2013-03-08 01:21:05,158 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [310a8b38] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:21:05,172 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [310a8b38] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3554ff19 >2013-03-08 01:21:05,172 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [310a8b38] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 01:21:05,172 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [310a8b38] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3554ff19 >2013-03-08 01:21:05,174 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [310a8b38] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 01:21:05,175 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [16409e5] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 01:21:05,184 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-10) [7ba61d9e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:21:05,185 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [7ba61d9e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2eda4f0f >2013-03-08 01:21:05,187 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [7ba61d9e] FINISH, SetVdsStatusVDSCommand, log id: 2eda4f0f >2013-03-08 01:21:05,193 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-10) [7ba61d9e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 01:21:05,215 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-10) [19cb5d6f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:21:05,218 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-10) [45069088] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:21:05,219 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-10) [45069088] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 01:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [489ff5bc] Autorecovering 1 hosts >2013-03-08 01:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [489ff5bc] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [652473e6] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [652473e6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [652473e6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7a9f8cdc >2013-03-08 01:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [652473e6] FINISH, SetVdsStatusVDSCommand, log id: 7a9f8cdc >2013-03-08 01:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [652473e6] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6aa3713b >2013-03-08 01:25:00,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [652473e6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7c93f0ff >2013-03-08 01:25:00,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [652473e6] FINISH, GetHardwareInfoVDSCommand, log id: 7c93f0ff >2013-03-08 01:25:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [ed12a73] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:25:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [2b11b881] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:25:00,093 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [2b11b881] FINISH, ActivateVdsVDSCommand, log id: 6aa3713b >2013-03-08 01:25:00,113 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [2b11b881] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:25:00,114 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-94) [2b11b881] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 01:25:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [2b11b881] Autorecovering 1 storage domains >2013-03-08 01:25:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [2b11b881] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 01:25:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [15c4844b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:25:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [15c4844b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:25 AM >2013-03-08 01:25:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [15c4844b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:25 AM >2013-03-08 01:25:02,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [27d79d50] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5044702d >2013-03-08 01:25:02,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [27d79d50] FINISH, GetHardwareInfoVDSCommand, log id: 5044702d >2013-03-08 01:25:02,468 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-65) [19e4d8ab] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 01:25:02,472 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-65) [1c91f4b1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 01:25:02,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 13f925be >2013-03-08 01:26:06,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 01:26:06,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:26:06,560 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 01:26:06,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 13f925be >2013-03-08 01:26:06,565 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [287b467f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:26:06,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [287b467f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 56962c46 >2013-03-08 01:26:06,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [287b467f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 01:26:06,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [287b467f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 56962c46 >2013-03-08 01:26:06,582 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [287b467f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 01:26:06,582 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [1c91f4b1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 01:26:06,599 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-65) [3b14be7e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:26:06,600 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [3b14be7e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 45ee739f >2013-03-08 01:26:06,602 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [3b14be7e] FINISH, SetVdsStatusVDSCommand, log id: 45ee739f >2013-03-08 01:26:06,607 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-65) [3b14be7e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 01:26:06,627 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-65) [414d134b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:26:06,630 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-65) [622b808a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:26:06,631 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-65) [622b808a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 01:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [3ae408a5] Autorecovering 1 hosts >2013-03-08 01:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [3ae408a5] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [24fe0ea9] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [24fe0ea9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [24fe0ea9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 41f33606 >2013-03-08 01:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [24fe0ea9] FINISH, SetVdsStatusVDSCommand, log id: 41f33606 >2013-03-08 01:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-68) [24fe0ea9] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2cc4ff1 >2013-03-08 01:30:00,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [24fe0ea9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5c52bb98 >2013-03-08 01:30:00,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [24fe0ea9] FINISH, GetHardwareInfoVDSCommand, log id: 5c52bb98 >2013-03-08 01:30:00,185 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-68) [679bab0d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:30:00,187 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-68) [4536d332] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:30:00,189 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-68) [4536d332] FINISH, ActivateVdsVDSCommand, log id: 2cc4ff1 >2013-03-08 01:30:00,209 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [4536d332] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:30:00,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-68) [4536d332] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 01:30:00,211 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [4536d332] Autorecovering 1 storage domains >2013-03-08 01:30:00,211 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [4536d332] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 01:30:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [ddd9e11] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:30:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [ddd9e11] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:30 AM >2013-03-08 01:30:00,215 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [ddd9e11] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:30 AM >2013-03-08 01:30:00,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [4c1eb652] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5d676971 >2013-03-08 01:30:00,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [4c1eb652] FINISH, GetHardwareInfoVDSCommand, log id: 5d676971 >2013-03-08 01:30:00,884 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-46) [1e9c0e9] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 01:30:00,888 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-46) [7e50bfe0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 01:30:00,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 10461b33 >2013-03-08 01:30:33,716 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-3) [d093265] Start refreshing all users data >2013-03-08 01:30:34,625 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-37) [2bd92ca5] Attempting to update VMs/Templates Ovf. >2013-03-08 01:30:34,626 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-37) [2bd92ca5] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 01:30:34,633 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-37) [2bd92ca5] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 01:30:34,633 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-37) [2bd92ca5] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 01:30:34,636 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-37) [2bd92ca5] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 01:30:34,636 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-37) [2bd92ca5] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 01:30:34,637 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-37) [2bd92ca5] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 01:31:04,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 01:31:04,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:31:04,883 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 01:31:04,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 10461b33 >2013-03-08 01:31:04,887 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [57deb607] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:31:04,903 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [57deb607] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: ca462a9 >2013-03-08 01:31:04,903 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [57deb607] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 01:31:04,903 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [57deb607] FINISH, MarkPoolInReconstructModeVDSCommand, log id: ca462a9 >2013-03-08 01:31:04,905 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [57deb607] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 01:31:04,905 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [7e50bfe0] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 01:31:04,916 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-46) [15d0fc3f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:31:04,917 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [15d0fc3f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4be8ebb8 >2013-03-08 01:31:04,919 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [15d0fc3f] FINISH, SetVdsStatusVDSCommand, log id: 4be8ebb8 >2013-03-08 01:31:04,924 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-46) [15d0fc3f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 01:31:04,945 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-46) [1f6bd98] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:31:04,948 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-46) [29ec8a6a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:31:04,949 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-46) [29ec8a6a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 01:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [34c22d61] Autorecovering 1 hosts >2013-03-08 01:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [34c22d61] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [15e07932] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [15e07932] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [15e07932] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 75643126 >2013-03-08 01:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [15e07932] FINISH, SetVdsStatusVDSCommand, log id: 75643126 >2013-03-08 01:35:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-87) [15e07932] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 20f5726d >2013-03-08 01:35:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [15e07932] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3eef83f9 >2013-03-08 01:35:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [15e07932] FINISH, GetHardwareInfoVDSCommand, log id: 3eef83f9 >2013-03-08 01:35:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [36ca46ec] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:35:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [53581832] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:35:00,206 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-87) [53581832] FINISH, ActivateVdsVDSCommand, log id: 20f5726d >2013-03-08 01:35:00,225 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [53581832] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:35:00,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [53581832] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 01:35:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [53581832] Autorecovering 1 storage domains >2013-03-08 01:35:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [53581832] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 01:35:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [4f61cbc8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:35:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [4f61cbc8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:35 AM >2013-03-08 01:35:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [4f61cbc8] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:35 AM >2013-03-08 01:35:02,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [15c4844b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 773b44 >2013-03-08 01:35:02,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [15c4844b] FINISH, GetHardwareInfoVDSCommand, log id: 773b44 >2013-03-08 01:35:02,198 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-94) [3ab9461f] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 01:35:02,203 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-94) [43b1a007] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 01:35:02,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7ceadada >2013-03-08 01:36:05,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 01:36:05,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:36:05,179 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 01:36:05,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 7ceadada >2013-03-08 01:36:05,183 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [76079689] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:36:05,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [76079689] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7f6f24 >2013-03-08 01:36:05,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [76079689] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 01:36:05,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [76079689] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7f6f24 >2013-03-08 01:36:05,203 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [76079689] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 01:36:05,204 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [43b1a007] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 01:36:05,222 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-94) [70500f77] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:36:05,224 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [70500f77] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 797ec0bd >2013-03-08 01:36:05,225 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [70500f77] FINISH, SetVdsStatusVDSCommand, log id: 797ec0bd >2013-03-08 01:36:05,234 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-94) [70500f77] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 01:36:05,267 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [112c8bec] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:36:05,269 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [2e7f5c74] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:36:05,271 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-94) [2e7f5c74] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 01:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [59cf7b26] Autorecovering 1 hosts >2013-03-08 01:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [59cf7b26] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [313d8ec2] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [313d8ec2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:40:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [313d8ec2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2798fa93 >2013-03-08 01:40:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [313d8ec2] FINISH, SetVdsStatusVDSCommand, log id: 2798fa93 >2013-03-08 01:40:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-33) [313d8ec2] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: ef3a309 >2013-03-08 01:40:00,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [313d8ec2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 43fd0e08 >2013-03-08 01:40:00,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [313d8ec2] FINISH, GetHardwareInfoVDSCommand, log id: 43fd0e08 >2013-03-08 01:40:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-33) [775901fc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:40:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-33) [320a006] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:40:00,206 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-33) [320a006] FINISH, ActivateVdsVDSCommand, log id: ef3a309 >2013-03-08 01:40:00,226 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [320a006] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:40:00,227 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-33) [320a006] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 01:40:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [320a006] Autorecovering 1 storage domains >2013-03-08 01:40:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [320a006] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 01:40:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [1d4b6bc2] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:40:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [1d4b6bc2] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:40 AM >2013-03-08 01:40:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [1d4b6bc2] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:40 AM >2013-03-08 01:40:02,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [283308f8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 20023589 >2013-03-08 01:40:02,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [283308f8] FINISH, GetHardwareInfoVDSCommand, log id: 20023589 >2013-03-08 01:40:02,521 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-11) [57ecbf21] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 01:40:02,524 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-11) [46f41b13] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 01:40:02,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2a24f09c >2013-03-08 01:41:06,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 01:41:06,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:41:06,806 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 01:41:06,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 2a24f09c >2013-03-08 01:41:06,810 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [77c4d73b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:41:06,825 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [77c4d73b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 113af125 >2013-03-08 01:41:06,825 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [77c4d73b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 01:41:06,825 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [77c4d73b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 113af125 >2013-03-08 01:41:06,827 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [77c4d73b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 01:41:06,828 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [46f41b13] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 01:41:06,837 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-11) [43e94e53] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:41:06,842 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [43e94e53] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 513daf83 >2013-03-08 01:41:06,843 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [43e94e53] FINISH, SetVdsStatusVDSCommand, log id: 513daf83 >2013-03-08 01:41:06,848 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-11) [43e94e53] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 01:41:06,871 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-11) [5eac0db6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:41:06,879 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-11) [78dbf18] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:41:06,881 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-11) [78dbf18] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 01:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [4ae1c7cf] Autorecovering 1 hosts >2013-03-08 01:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [4ae1c7cf] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [31c641dd] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [31c641dd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [31c641dd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3a583be4 >2013-03-08 01:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [31c641dd] FINISH, SetVdsStatusVDSCommand, log id: 3a583be4 >2013-03-08 01:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [31c641dd] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 63d25217 >2013-03-08 01:45:00,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [31c641dd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6f27531d >2013-03-08 01:45:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [31c641dd] FINISH, GetHardwareInfoVDSCommand, log id: 6f27531d >2013-03-08 01:45:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [6faedd0a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:45:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [652709c2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:45:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [652709c2] FINISH, ActivateVdsVDSCommand, log id: 63d25217 >2013-03-08 01:45:00,219 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [652709c2] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:45:00,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [652709c2] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 01:45:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [652709c2] Autorecovering 1 storage domains >2013-03-08 01:45:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [652709c2] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 01:45:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [15f9bed0] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:45:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [15f9bed0] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:45 AM >2013-03-08 01:45:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [15f9bed0] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:45 AM >2013-03-08 01:45:01,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [db405a5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 76db3be4 >2013-03-08 01:45:01,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [db405a5] FINISH, GetHardwareInfoVDSCommand, log id: 76db3be4 >2013-03-08 01:45:01,153 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-93) [21871ac6] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 01:45:01,156 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-93) [313cd1c0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 01:45:01,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2f1d9707 >2013-03-08 01:46:05,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 01:46:05,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:46:05,139 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 01:46:05,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 2f1d9707 >2013-03-08 01:46:05,143 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [4aa4188b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:46:05,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4aa4188b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3135116d >2013-03-08 01:46:05,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [4aa4188b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 01:46:05,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4aa4188b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3135116d >2013-03-08 01:46:05,160 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [4aa4188b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 01:46:05,161 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [313cd1c0] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 01:46:05,172 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-93) [21cfeb98] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:46:05,174 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [21cfeb98] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: ba22c93 >2013-03-08 01:46:05,175 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [21cfeb98] FINISH, SetVdsStatusVDSCommand, log id: ba22c93 >2013-03-08 01:46:05,181 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-93) [21cfeb98] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 01:46:05,202 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [7581a780] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:46:05,205 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [51b03692] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:46:05,206 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-93) [51b03692] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 01:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [804dd00] Autorecovering 1 hosts >2013-03-08 01:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [804dd00] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-15) [42ef9030] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-15) [42ef9030] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [42ef9030] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 24b7d6a4 >2013-03-08 01:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [42ef9030] FINISH, SetVdsStatusVDSCommand, log id: 24b7d6a4 >2013-03-08 01:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-15) [42ef9030] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: ee0dc1f >2013-03-08 01:50:00,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [42ef9030] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5137a53c >2013-03-08 01:50:00,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [42ef9030] FINISH, GetHardwareInfoVDSCommand, log id: 5137a53c >2013-03-08 01:50:00,416 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-15) [30d27a66] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:50:00,418 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-15) [12364e89] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:50:00,419 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-15) [12364e89] FINISH, ActivateVdsVDSCommand, log id: ee0dc1f >2013-03-08 01:50:00,439 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-15) [12364e89] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:50:00,440 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-15) [12364e89] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 01:50:00,441 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [12364e89] Autorecovering 1 storage domains >2013-03-08 01:50:00,441 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [12364e89] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 01:50:00,442 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-15) [41d2a2b8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:50:00,442 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-15) [41d2a2b8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:50 AM >2013-03-08 01:50:00,445 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-15) [41d2a2b8] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:50 AM >2013-03-08 01:50:02,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [6d273a42] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7bceb92b >2013-03-08 01:50:02,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [6d273a42] FINISH, GetHardwareInfoVDSCommand, log id: 7bceb92b >2013-03-08 01:50:02,451 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-2) [4a6cffcb] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 01:50:02,455 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-2) [4edbcc69] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 01:50:02,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5118ee38 >2013-03-08 01:51:07,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 01:51:07,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:51:07,430 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 01:51:07,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 5118ee38 >2013-03-08 01:51:07,434 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [23dddf06] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:51:07,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [23dddf06] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: fb93d7b >2013-03-08 01:51:07,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [23dddf06] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 01:51:07,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [23dddf06] FINISH, MarkPoolInReconstructModeVDSCommand, log id: fb93d7b >2013-03-08 01:51:07,451 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [23dddf06] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 01:51:07,451 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [4edbcc69] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 01:51:07,461 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-2) [1d68a5c2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:51:07,463 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [1d68a5c2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5af09374 >2013-03-08 01:51:07,464 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [1d68a5c2] FINISH, SetVdsStatusVDSCommand, log id: 5af09374 >2013-03-08 01:51:07,470 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [1d68a5c2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 01:51:07,490 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [5ccf4acc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:51:07,493 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [4114dfcd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:51:07,494 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-2) [4114dfcd] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 01:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [6930a36d] Autorecovering 1 hosts >2013-03-08 01:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [6930a36d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [3ba8caed] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [3ba8caed] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [3ba8caed] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1e95ffd2 >2013-03-08 01:55:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [3ba8caed] FINISH, SetVdsStatusVDSCommand, log id: 1e95ffd2 >2013-03-08 01:55:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-88) [3ba8caed] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 16ed9e97 >2013-03-08 01:55:00,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [3ba8caed] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 901fd38 >2013-03-08 01:55:00,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [3ba8caed] FINISH, GetHardwareInfoVDSCommand, log id: 901fd38 >2013-03-08 01:55:00,098 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-88) [2914e8cc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:55:00,100 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-88) [47ef33a9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:55:00,101 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-88) [47ef33a9] FINISH, ActivateVdsVDSCommand, log id: 16ed9e97 >2013-03-08 01:55:00,120 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [47ef33a9] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 01:55:00,121 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-88) [47ef33a9] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 01:55:00,123 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [47ef33a9] Autorecovering 1 storage domains >2013-03-08 01:55:00,123 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [47ef33a9] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 01:55:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [1b3bfb6d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:55:00,124 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [1b3bfb6d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:55 AM >2013-03-08 01:55:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [1b3bfb6d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:55 AM >2013-03-08 01:55:01,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [50b7f5f1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 42734be >2013-03-08 01:55:01,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [50b7f5f1] FINISH, GetHardwareInfoVDSCommand, log id: 42734be >2013-03-08 01:55:01,744 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-49) [364a2452] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 01:55:01,748 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-49) [3eadae8e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 01:55:01,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3f4027ff >2013-03-08 01:56:05,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 01:56:05,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 01:56:05,795 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 01:56:05,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 3f4027ff >2013-03-08 01:56:05,798 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [1353378d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 01:56:05,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [1353378d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3abbd921 >2013-03-08 01:56:05,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [1353378d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 01:56:05,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [1353378d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3abbd921 >2013-03-08 01:56:05,815 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [1353378d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 01:56:05,815 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [3eadae8e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 01:56:05,825 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-49) [4748cd05] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:56:05,827 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [4748cd05] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 52e7c26a >2013-03-08 01:56:05,828 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [4748cd05] FINISH, SetVdsStatusVDSCommand, log id: 52e7c26a >2013-03-08 01:56:05,834 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-49) [4748cd05] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 01:56:05,864 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-49) [4c6e84cf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:56:05,867 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-49) [206eb1f0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 01:56:05,868 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) [206eb1f0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 02:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [d34ddb5] Autorecovering 1 hosts >2013-03-08 02:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [d34ddb5] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [5a70b128] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [5a70b128] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [5a70b128] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5e2f9cad >2013-03-08 02:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [5a70b128] FINISH, SetVdsStatusVDSCommand, log id: 5e2f9cad >2013-03-08 02:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-20) [5a70b128] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 494a8beb >2013-03-08 02:00:00,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [2658832] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2206cebb >2013-03-08 02:00:00,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [2658832] FINISH, GetHardwareInfoVDSCommand, log id: 2206cebb >2013-03-08 02:00:00,429 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-4) [3614d0fa] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 02:00:00,433 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-4) [71534e27] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 02:00:00,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 64fce4c1 >2013-03-08 02:01:05,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 02:01:05,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:01:05,439 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 02:01:05,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 64fce4c1 >2013-03-08 02:01:05,443 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [6cc07754] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:01:05,458 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [6cc07754] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 65bdfa81 >2013-03-08 02:01:05,458 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [6cc07754] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 02:01:05,458 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [6cc07754] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 65bdfa81 >2013-03-08 02:01:05,460 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [6cc07754] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 02:01:05,461 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [71534e27] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 02:01:05,471 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-4) [1ab3c01a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:01:05,472 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [1ab3c01a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 462d067d >2013-03-08 02:01:05,474 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [1ab3c01a] FINISH, SetVdsStatusVDSCommand, log id: 462d067d >2013-03-08 02:01:05,479 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-4) [1ab3c01a] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 02:01:05,499 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-4) [580e38be] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:01:05,501 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-4) [52ed2d33] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:01:05,503 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-4) [52ed2d33] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 02:01:05,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [5a70b128] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: b301cec >2013-03-08 02:01:05,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [5a70b128] FINISH, GetHardwareInfoVDSCommand, log id: b301cec >2013-03-08 02:01:05,581 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-20) [168fc46c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:01:05,583 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-20) [3f821ac3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:01:05,585 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-20) [3f821ac3] FINISH, ActivateVdsVDSCommand, log id: 494a8beb >2013-03-08 02:01:05,604 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [3f821ac3] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:01:05,605 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-20) [3f821ac3] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 02:01:05,606 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [3f821ac3] Autorecovering 1 storage domains >2013-03-08 02:01:05,606 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [3f821ac3] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 02:01:05,607 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [62df4fdf] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:01:05,607 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [62df4fdf] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:01 AM >2013-03-08 02:01:05,610 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [62df4fdf] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:01 AM >2013-03-08 02:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-37) [2bd92ca5] Autorecovering 1 hosts >2013-03-08 02:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-37) [2bd92ca5] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-37) [2e4ff7df] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-37) [2e4ff7df] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [2e4ff7df] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 67a1fc66 >2013-03-08 02:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [2e4ff7df] FINISH, SetVdsStatusVDSCommand, log id: 67a1fc66 >2013-03-08 02:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-37) [2e4ff7df] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 11b9da97 >2013-03-08 02:05:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [2e4ff7df] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5b135a6c >2013-03-08 02:05:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [2e4ff7df] FINISH, GetHardwareInfoVDSCommand, log id: 5b135a6c >2013-03-08 02:05:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-37) [76243abe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:05:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-37) [f657c7d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:05:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-37) [f657c7d] FINISH, ActivateVdsVDSCommand, log id: 11b9da97 >2013-03-08 02:05:00,112 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-37) [f657c7d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:05:00,113 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-37) [f657c7d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 02:05:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-37) [f657c7d] Autorecovering 1 storage domains >2013-03-08 02:05:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-37) [f657c7d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 02:05:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-37) [3663092] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:05:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-37) [3663092] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:05 AM >2013-03-08 02:05:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-37) [3663092] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:05 AM >2013-03-08 02:05:02,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [1b3bfb6d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 92aba3 >2013-03-08 02:05:02,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [1b3bfb6d] FINISH, GetHardwareInfoVDSCommand, log id: 92aba3 >2013-03-08 02:05:02,751 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-88) [77275c53] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 02:05:02,754 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-88) [37df96e6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 02:05:02,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 60b35de >2013-03-08 02:06:05,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 02:06:05,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:06:05,755 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 02:06:05,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 60b35de >2013-03-08 02:06:05,759 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [67096f29] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:06:05,773 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [67096f29] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 303bfc88 >2013-03-08 02:06:05,773 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [67096f29] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 02:06:05,773 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [67096f29] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 303bfc88 >2013-03-08 02:06:05,775 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [67096f29] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 02:06:05,776 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [37df96e6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 02:06:05,785 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-88) [2e58fcce] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:06:05,787 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [2e58fcce] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7645bd50 >2013-03-08 02:06:05,788 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [2e58fcce] FINISH, SetVdsStatusVDSCommand, log id: 7645bd50 >2013-03-08 02:06:05,793 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-88) [2e58fcce] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 02:06:05,813 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-88) [68d26b4b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:06:05,816 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-88) [afe113b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:06:05,817 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-88) [afe113b] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 02:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [39a060ae] Autorecovering 1 hosts >2013-03-08 02:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [39a060ae] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-25) [30e36665] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-25) [30e36665] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [30e36665] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 362a5f32 >2013-03-08 02:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [30e36665] FINISH, SetVdsStatusVDSCommand, log id: 362a5f32 >2013-03-08 02:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-25) [30e36665] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: def29f3 >2013-03-08 02:10:00,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [30e36665] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3bb76ed2 >2013-03-08 02:10:00,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [30e36665] FINISH, GetHardwareInfoVDSCommand, log id: 3bb76ed2 >2013-03-08 02:10:00,402 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-25) [33fe698d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:10:00,405 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-25) [1cb01844] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:10:00,406 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-25) [1cb01844] FINISH, ActivateVdsVDSCommand, log id: def29f3 >2013-03-08 02:10:00,426 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-25) [1cb01844] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:10:00,427 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-25) [1cb01844] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 02:10:00,428 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [1cb01844] Autorecovering 1 storage domains >2013-03-08 02:10:00,428 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [1cb01844] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 02:10:00,429 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-25) [7daa393a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:10:00,429 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-25) [7daa393a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:10 AM >2013-03-08 02:10:00,432 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-25) [7daa393a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:10 AM >2013-03-08 02:10:03,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [69e2dae5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 99a037a >2013-03-08 02:10:03,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [69e2dae5] FINISH, GetHardwareInfoVDSCommand, log id: 99a037a >2013-03-08 02:10:03,066 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-57) [2608304f] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 02:10:03,069 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-57) [19fc965] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 02:10:03,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 38b8005a >2013-03-08 02:11:09,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 02:11:09,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:11:09,077 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 02:11:09,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 38b8005a >2013-03-08 02:11:09,081 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [524ebe15] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:11:09,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [524ebe15] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: fa34c1e >2013-03-08 02:11:09,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [524ebe15] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 02:11:09,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [524ebe15] FINISH, MarkPoolInReconstructModeVDSCommand, log id: fa34c1e >2013-03-08 02:11:09,102 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [524ebe15] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 02:11:09,103 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [19fc965] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 02:11:09,112 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-57) [41f31a72] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:11:09,114 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [41f31a72] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 775eb2df >2013-03-08 02:11:09,115 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [41f31a72] FINISH, SetVdsStatusVDSCommand, log id: 775eb2df >2013-03-08 02:11:09,120 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-57) [41f31a72] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 02:11:09,141 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-57) [a801fed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:11:09,144 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-57) [dbdac2d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:11:09,145 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-57) [dbdac2d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 02:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) [5c32a01d] Autorecovering 1 hosts >2013-03-08 02:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) [5c32a01d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-34) [25198919] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-34) [25198919] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [25198919] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 559381e0 >2013-03-08 02:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [25198919] FINISH, SetVdsStatusVDSCommand, log id: 559381e0 >2013-03-08 02:15:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-34) [25198919] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3a848e4e >2013-03-08 02:15:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [25198919] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4794c9d9 >2013-03-08 02:15:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [25198919] FINISH, GetHardwareInfoVDSCommand, log id: 4794c9d9 >2013-03-08 02:15:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-34) [31a74c1c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:15:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-34) [17794cfc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:15:00,095 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-34) [17794cfc] FINISH, ActivateVdsVDSCommand, log id: 3a848e4e >2013-03-08 02:15:00,114 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-34) [17794cfc] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:15:00,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-34) [17794cfc] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 02:15:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) [17794cfc] Autorecovering 1 storage domains >2013-03-08 02:15:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) [17794cfc] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 02:15:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-34) [36e1c345] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:15:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-34) [36e1c345] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:15 AM >2013-03-08 02:15:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-34) [36e1c345] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:15 AM >2013-03-08 02:15:00,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [68207617] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 210b3139 >2013-03-08 02:15:00,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [68207617] FINISH, GetHardwareInfoVDSCommand, log id: 210b3139 >2013-03-08 02:15:00,385 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-77) [320001c5] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 02:15:00,389 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-77) [52618bf2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 02:15:00,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 428cc011 >2013-03-08 02:16:04,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 02:16:04,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:16:04,419 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 02:16:04,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 428cc011 >2013-03-08 02:16:04,423 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [7076e43d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:16:04,438 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [7076e43d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1d508117 >2013-03-08 02:16:04,438 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [7076e43d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 02:16:04,438 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [7076e43d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1d508117 >2013-03-08 02:16:04,440 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [7076e43d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 02:16:04,441 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [52618bf2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 02:16:04,451 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-77) [6101e18a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:16:04,452 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [6101e18a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 53a341fe >2013-03-08 02:16:04,453 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [6101e18a] FINISH, SetVdsStatusVDSCommand, log id: 53a341fe >2013-03-08 02:16:04,459 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-77) [6101e18a] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 02:16:04,480 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-77) [3fa5ea85] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:16:04,484 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-77) [99e81db] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:16:04,485 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-77) [99e81db] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 02:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [11e45fdf] Autorecovering 1 hosts >2013-03-08 02:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [11e45fdf] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-38) [5ff5e742] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-38) [5ff5e742] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [5ff5e742] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 30673e8c >2013-03-08 02:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [5ff5e742] FINISH, SetVdsStatusVDSCommand, log id: 30673e8c >2013-03-08 02:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-38) [5ff5e742] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6ed211d2 >2013-03-08 02:20:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [5ff5e742] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 755d3f18 >2013-03-08 02:20:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [5ff5e742] FINISH, GetHardwareInfoVDSCommand, log id: 755d3f18 >2013-03-08 02:20:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-38) [31e4929e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:20:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-38) [4ed83e2f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:20:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-38) [4ed83e2f] FINISH, ActivateVdsVDSCommand, log id: 6ed211d2 >2013-03-08 02:20:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-38) [4ed83e2f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:20:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-38) [4ed83e2f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 02:20:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [4ed83e2f] Autorecovering 1 storage domains >2013-03-08 02:20:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [4ed83e2f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 02:20:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-38) [7e3606b9] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:20:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-38) [7e3606b9] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:20 AM >2013-03-08 02:20:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-38) [7e3606b9] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:20 AM >2013-03-08 02:20:01,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [3b606a7f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 25fc9b3b >2013-03-08 02:20:01,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [3b606a7f] FINISH, GetHardwareInfoVDSCommand, log id: 25fc9b3b >2013-03-08 02:20:01,740 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-16) [2ee380db] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 02:20:01,743 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-16) [474f4d8b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 02:20:01,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 42001030 >2013-03-08 02:21:07,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 02:21:07,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:21:07,745 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 02:21:07,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 42001030 >2013-03-08 02:21:07,749 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [4b5dddcc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:21:07,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4b5dddcc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 18f02ade >2013-03-08 02:21:07,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [4b5dddcc] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 02:21:07,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4b5dddcc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 18f02ade >2013-03-08 02:21:07,773 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [474f4d8b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 02:21:07,783 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-16) [63d61a3e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:21:07,807 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [63d61a3e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 23faa46c >2013-03-08 02:21:07,812 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [63d61a3e] FINISH, SetVdsStatusVDSCommand, log id: 23faa46c >2013-03-08 02:21:07,820 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-16) [63d61a3e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 02:21:07,853 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-16) [1606258c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:21:07,856 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-16) [2e8c04d6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:21:07,858 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-16) [2e8c04d6] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 02:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [36575662] Autorecovering 1 hosts >2013-03-08 02:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [36575662] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-67) [8c74fa] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-67) [8c74fa] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:25:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [8c74fa] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 40056dcc >2013-03-08 02:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [8c74fa] FINISH, SetVdsStatusVDSCommand, log id: 40056dcc >2013-03-08 02:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-67) [8c74fa] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5e5dc364 >2013-03-08 02:25:00,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [8c74fa] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 537aaa24 >2013-03-08 02:25:00,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [8c74fa] FINISH, GetHardwareInfoVDSCommand, log id: 537aaa24 >2013-03-08 02:25:00,402 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-67) [f4e62bd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:25:00,405 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-67) [26688f40] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:25:00,406 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-67) [26688f40] FINISH, ActivateVdsVDSCommand, log id: 5e5dc364 >2013-03-08 02:25:00,425 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-67) [26688f40] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:25:00,426 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-67) [26688f40] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 02:25:00,428 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [26688f40] Autorecovering 1 storage domains >2013-03-08 02:25:00,428 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [26688f40] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 02:25:00,428 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-67) [6ffac8dc] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:25:00,428 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-67) [6ffac8dc] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:25 AM >2013-03-08 02:25:00,431 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-67) [6ffac8dc] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:25 AM >2013-03-08 02:25:02,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-71) [4d98a416] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4f0200cb >2013-03-08 02:25:02,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-71) [4d98a416] FINISH, GetHardwareInfoVDSCommand, log id: 4f0200cb >2013-03-08 02:25:02,095 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-71) [684f0976] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 02:25:02,099 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-71) [17621560] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 02:25:02,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2c0fc3c5 >2013-03-08 02:26:07,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 02:26:07,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:26:07,126 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 02:26:07,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 2c0fc3c5 >2013-03-08 02:26:07,130 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [4f88d600] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:26:07,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4f88d600] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 64b64e8e >2013-03-08 02:26:07,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [4f88d600] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 02:26:07,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4f88d600] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 64b64e8e >2013-03-08 02:26:07,147 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [4f88d600] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 02:26:07,147 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [17621560] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 02:26:07,157 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-71) [607f2ff8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:26:07,159 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-71) [607f2ff8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: d648b6f >2013-03-08 02:26:07,160 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-71) [607f2ff8] FINISH, SetVdsStatusVDSCommand, log id: d648b6f >2013-03-08 02:26:07,166 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-71) [607f2ff8] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 02:26:07,187 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-71) [6c9d55de] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:26:07,189 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-71) [3640068] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:26:07,191 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-71) [3640068] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 02:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) Autorecovering 1 hosts >2013-03-08 02:30:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [2c016b69] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [2c016b69] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [2c016b69] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 921abe8 >2013-03-08 02:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [2c016b69] FINISH, SetVdsStatusVDSCommand, log id: 921abe8 >2013-03-08 02:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [2c016b69] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 29e3c932 >2013-03-08 02:30:00,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [2c016b69] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7b009006 >2013-03-08 02:30:00,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [2c016b69] FINISH, GetHardwareInfoVDSCommand, log id: 7b009006 >2013-03-08 02:30:00,101 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-42) [1612e2f2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:30:00,103 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-42) [4b7a35b0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:30:00,105 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [4b7a35b0] FINISH, ActivateVdsVDSCommand, log id: 29e3c932 >2013-03-08 02:30:00,124 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [4b7a35b0] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:30:00,125 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-42) [4b7a35b0] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 02:30:00,126 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [4b7a35b0] Autorecovering 1 storage domains >2013-03-08 02:30:00,126 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [4b7a35b0] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 02:30:00,127 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [112a9c27] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:30:00,127 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [112a9c27] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:30 AM >2013-03-08 02:30:00,130 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [112a9c27] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:30 AM >2013-03-08 02:30:01,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [3a39221f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5cc99d2e >2013-03-08 02:30:01,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [3a39221f] FINISH, GetHardwareInfoVDSCommand, log id: 5cc99d2e >2013-03-08 02:30:01,438 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-75) [50197950] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 02:30:01,442 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-75) [cea651c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 02:30:01,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 158921bd >2013-03-08 02:30:33,719 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-5) [434cb3a8] Start refreshing all users data >2013-03-08 02:30:34,638 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-49) [206eb1f0] Attempting to update VMs/Templates Ovf. >2013-03-08 02:30:34,639 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-49) [206eb1f0] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 02:30:34,640 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-49) [206eb1f0] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 02:30:34,640 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-49) [206eb1f0] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 02:30:34,641 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-49) [206eb1f0] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 02:30:34,641 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-49) [206eb1f0] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 02:30:34,642 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-49) [206eb1f0] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 02:31:05,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 02:31:05,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:31:05,458 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 02:31:05,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 158921bd >2013-03-08 02:31:05,463 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [2fd6dfb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:31:05,483 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [2fd6dfb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2d6e6b06 >2013-03-08 02:31:05,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [2fd6dfb] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 02:31:05,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [2fd6dfb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2d6e6b06 >2013-03-08 02:31:05,486 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [2fd6dfb] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 02:31:05,486 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [cea651c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 02:31:05,496 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-75) [307c9b25] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:31:05,497 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [307c9b25] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 60873dd9 >2013-03-08 02:31:05,498 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [307c9b25] FINISH, SetVdsStatusVDSCommand, log id: 60873dd9 >2013-03-08 02:31:05,503 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-75) [307c9b25] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 02:31:05,523 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [650739e3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:31:05,526 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [bbb9476] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:31:05,527 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-75) [bbb9476] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 02:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [467d1c57] Autorecovering 1 hosts >2013-03-08 02:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [467d1c57] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:35:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [1af280bd] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:35:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [1af280bd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:35:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [1af280bd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 73f19ec7 >2013-03-08 02:35:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [1af280bd] FINISH, SetVdsStatusVDSCommand, log id: 73f19ec7 >2013-03-08 02:35:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [1af280bd] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 8e52658 >2013-03-08 02:35:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [1af280bd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4284363b >2013-03-08 02:35:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [1af280bd] FINISH, GetHardwareInfoVDSCommand, log id: 4284363b >2013-03-08 02:35:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-84) [111aaea9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:35:00,206 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-84) [6bec1228] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:35:00,208 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [6bec1228] FINISH, ActivateVdsVDSCommand, log id: 8e52658 >2013-03-08 02:35:00,230 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [6bec1228] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:35:00,231 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-84) [6bec1228] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 02:35:00,233 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [6bec1228] Autorecovering 1 storage domains >2013-03-08 02:35:00,233 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [6bec1228] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 02:35:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [16a2746f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:35:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [16a2746f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:35 AM >2013-03-08 02:35:00,240 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [16a2746f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:35 AM >2013-03-08 02:35:02,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [1899bf25] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 11c1c108 >2013-03-08 02:35:02,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [1899bf25] FINISH, GetHardwareInfoVDSCommand, log id: 11c1c108 >2013-03-08 02:35:02,775 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-91) [7fad602d] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 02:35:02,778 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-91) [321a6ddb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 02:35:02,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 483b934d >2013-03-08 02:36:08,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 02:36:08,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:36:08,771 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 02:36:08,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 483b934d >2013-03-08 02:36:08,775 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [5867334c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:36:08,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [5867334c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7e5df3e >2013-03-08 02:36:08,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [5867334c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 02:36:08,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [5867334c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7e5df3e >2013-03-08 02:36:08,792 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [5867334c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 02:36:08,792 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [321a6ddb] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 02:36:08,804 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-91) [168c389d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:36:08,806 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [168c389d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7dfb7b8f >2013-03-08 02:36:08,807 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [168c389d] FINISH, SetVdsStatusVDSCommand, log id: 7dfb7b8f >2013-03-08 02:36:08,813 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-91) [168c389d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 02:36:08,834 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-91) [5070da38] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:36:08,836 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-91) [1fa2973c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:36:08,838 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-91) [1fa2973c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 02:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [2b741d8c] Autorecovering 1 hosts >2013-03-08 02:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [2b741d8c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [7a4dbed9] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [7a4dbed9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [7a4dbed9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7ec2d732 >2013-03-08 02:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [7a4dbed9] FINISH, SetVdsStatusVDSCommand, log id: 7ec2d732 >2013-03-08 02:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [7a4dbed9] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3074a44d >2013-03-08 02:40:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [7a4dbed9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1670381 >2013-03-08 02:40:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [7a4dbed9] FINISH, GetHardwareInfoVDSCommand, log id: 1670381 >2013-03-08 02:40:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [16df7658] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:40:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [32fc5e1e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:40:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [32fc5e1e] FINISH, ActivateVdsVDSCommand, log id: 3074a44d >2013-03-08 02:40:00,222 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [32fc5e1e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:40:00,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-52) [32fc5e1e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 02:40:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [32fc5e1e] Autorecovering 1 storage domains >2013-03-08 02:40:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [32fc5e1e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 02:40:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [151938f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:40:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [151938f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:40 AM >2013-03-08 02:40:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [151938f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:40 AM >2013-03-08 02:40:03,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [112a9c27] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4053a32b >2013-03-08 02:40:03,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [112a9c27] FINISH, GetHardwareInfoVDSCommand, log id: 4053a32b >2013-03-08 02:40:03,082 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-42) [2da428d1] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 02:40:03,085 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-42) [13f70561] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 02:40:03,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 61850e4e >2013-03-08 02:41:07,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 02:41:07,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:41:07,082 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 02:41:07,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 61850e4e >2013-03-08 02:41:07,085 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [a3e12ac] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:41:07,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [a3e12ac] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 638f53ee >2013-03-08 02:41:07,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [a3e12ac] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 02:41:07,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [a3e12ac] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 638f53ee >2013-03-08 02:41:07,102 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [a3e12ac] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 02:41:07,102 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [13f70561] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 02:41:07,112 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-42) [1c1b0d7c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:41:07,114 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [1c1b0d7c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4fdef901 >2013-03-08 02:41:07,115 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [1c1b0d7c] FINISH, SetVdsStatusVDSCommand, log id: 4fdef901 >2013-03-08 02:41:07,121 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-42) [1c1b0d7c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 02:41:07,142 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-42) [50c034f8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:41:07,144 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-42) [2e0e0b36] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:41:07,146 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-42) [2e0e0b36] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 02:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [11dfe85f] Autorecovering 1 hosts >2013-03-08 02:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [11dfe85f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [7986e68c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [7986e68c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [7986e68c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 790b6049 >2013-03-08 02:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [7986e68c] FINISH, SetVdsStatusVDSCommand, log id: 790b6049 >2013-03-08 02:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [7986e68c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 796a6823 >2013-03-08 02:45:00,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [7986e68c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7a957d2 >2013-03-08 02:45:00,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [7986e68c] FINISH, GetHardwareInfoVDSCommand, log id: 7a957d2 >2013-03-08 02:45:00,182 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [6574e9b6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:45:00,185 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [401ec6de] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:45:00,186 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [401ec6de] FINISH, ActivateVdsVDSCommand, log id: 796a6823 >2013-03-08 02:45:00,205 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [401ec6de] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:45:00,206 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [401ec6de] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 02:45:00,207 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [401ec6de] Autorecovering 1 storage domains >2013-03-08 02:45:00,208 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [401ec6de] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 02:45:00,208 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [715fa4a2] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:45:00,208 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [715fa4a2] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:45 AM >2013-03-08 02:45:00,211 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [715fa4a2] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:45 AM >2013-03-08 02:45:01,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [1022b3a1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3c530e67 >2013-03-08 02:45:01,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [1022b3a1] FINISH, GetHardwareInfoVDSCommand, log id: 3c530e67 >2013-03-08 02:45:01,393 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-90) [1a8d2cb3] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 02:45:01,396 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-90) [6659187a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 02:45:01,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6dcbd7a3 >2013-03-08 02:46:05,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 02:46:05,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:46:05,373 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 02:46:05,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 6dcbd7a3 >2013-03-08 02:46:05,377 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [369f44d1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:46:05,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [369f44d1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: d7cacd1 >2013-03-08 02:46:05,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [369f44d1] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 02:46:05,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [369f44d1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: d7cacd1 >2013-03-08 02:46:05,393 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [369f44d1] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 02:46:05,393 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [6659187a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 02:46:05,403 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-90) [2436b6ec] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:46:05,405 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [2436b6ec] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6136aed7 >2013-03-08 02:46:05,406 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [2436b6ec] FINISH, SetVdsStatusVDSCommand, log id: 6136aed7 >2013-03-08 02:46:05,412 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [2436b6ec] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 02:46:05,432 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-90) [58cddaea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:46:05,435 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-90) [158c1d87] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:46:05,436 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-90) [158c1d87] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 02:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [4114dfcd] Autorecovering 1 hosts >2013-03-08 02:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [4114dfcd] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [738b4d04] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [738b4d04] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [738b4d04] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 47184c77 >2013-03-08 02:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [738b4d04] FINISH, SetVdsStatusVDSCommand, log id: 47184c77 >2013-03-08 02:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-2) [738b4d04] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 761e79a5 >2013-03-08 02:50:00,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [738b4d04] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 181bc09e >2013-03-08 02:50:00,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [738b4d04] FINISH, GetHardwareInfoVDSCommand, log id: 181bc09e >2013-03-08 02:50:00,259 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [5e3f80df] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:50:00,262 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [9cbfeb6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:50:00,263 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-2) [9cbfeb6] FINISH, ActivateVdsVDSCommand, log id: 761e79a5 >2013-03-08 02:50:00,282 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [9cbfeb6] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:50:00,283 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [9cbfeb6] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 02:50:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [9cbfeb6] Autorecovering 1 storage domains >2013-03-08 02:50:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [9cbfeb6] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 02:50:00,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [70dd4adf] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:50:00,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [70dd4adf] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:50 AM >2013-03-08 02:50:00,289 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [70dd4adf] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:50 AM >2013-03-08 02:50:02,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [41ee8073] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7dad631a >2013-03-08 02:50:02,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [41ee8073] FINISH, GetHardwareInfoVDSCommand, log id: 7dad631a >2013-03-08 02:50:02,676 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-64) [52577b53] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 02:50:02,680 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-64) [1be8f18f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 02:50:02,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 15a2385b >2013-03-08 02:51:05,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 02:51:05,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:51:05,680 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 02:51:05,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 15a2385b >2013-03-08 02:51:05,684 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [1dd8bd6d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:51:05,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1dd8bd6d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1e237fbd >2013-03-08 02:51:05,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [1dd8bd6d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 02:51:05,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1dd8bd6d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1e237fbd >2013-03-08 02:51:05,700 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [1dd8bd6d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 02:51:05,700 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [1be8f18f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 02:51:05,711 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-64) [4d99ef26] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:51:05,712 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [4d99ef26] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6803f89f >2013-03-08 02:51:05,714 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [4d99ef26] FINISH, SetVdsStatusVDSCommand, log id: 6803f89f >2013-03-08 02:51:05,721 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-64) [4d99ef26] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 02:51:05,743 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-64) [71a69e84] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:51:05,746 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-64) [92bbf84] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:51:05,747 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-64) [92bbf84] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 02:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [51b03692] Autorecovering 1 hosts >2013-03-08 02:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [51b03692] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [780404dc] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [780404dc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [780404dc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6fd98e63 >2013-03-08 02:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [780404dc] FINISH, SetVdsStatusVDSCommand, log id: 6fd98e63 >2013-03-08 02:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-93) [780404dc] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6c73f5b8 >2013-03-08 02:55:00,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [780404dc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2f34c0e6 >2013-03-08 02:55:00,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [780404dc] FINISH, GetHardwareInfoVDSCommand, log id: 2f34c0e6 >2013-03-08 02:55:00,263 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [2c88d9ba] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:55:00,266 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [604460f5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:55:00,267 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-93) [604460f5] FINISH, ActivateVdsVDSCommand, log id: 6c73f5b8 >2013-03-08 02:55:00,287 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [604460f5] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 02:55:00,288 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-93) [604460f5] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 02:55:00,289 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [604460f5] Autorecovering 1 storage domains >2013-03-08 02:55:00,289 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [604460f5] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 02:55:00,290 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [57dbec90] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:55:00,290 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [57dbec90] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:55 AM >2013-03-08 02:55:00,293 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [57dbec90] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:55 AM >2013-03-08 02:55:02,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [9253b4f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1135662d >2013-03-08 02:55:02,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [9253b4f] FINISH, GetHardwareInfoVDSCommand, log id: 1135662d >2013-03-08 02:55:02,994 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-22) [7fa95b03] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 02:55:02,998 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-22) [40e417c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 02:55:03,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 63431b00 >2013-03-08 02:56:08,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 02:56:08,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 02:56:08,046 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 02:56:08,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 63431b00 >2013-03-08 02:56:08,050 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [69e96283] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 02:56:08,064 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [69e96283] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 663142e1 >2013-03-08 02:56:08,065 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [69e96283] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 02:56:08,065 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [69e96283] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 663142e1 >2013-03-08 02:56:08,067 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [69e96283] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 02:56:08,067 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [40e417c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 02:56:08,076 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-22) [14568370] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:56:08,078 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [14568370] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 18d207de >2013-03-08 02:56:08,079 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [14568370] FINISH, SetVdsStatusVDSCommand, log id: 18d207de >2013-03-08 02:56:08,085 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [14568370] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 02:56:08,105 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [3de53a53] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:56:08,108 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [785b3a66] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 02:56:08,109 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-22) [785b3a66] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 03:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [45054fea] Autorecovering 1 hosts >2013-03-08 03:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [45054fea] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-51) [618ec931] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-51) [618ec931] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [618ec931] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5729fdcc >2013-03-08 03:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [618ec931] FINISH, SetVdsStatusVDSCommand, log id: 5729fdcc >2013-03-08 03:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-51) [618ec931] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: b3f152 >2013-03-08 03:00:05,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [618ec931] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6d0bb0a1 >2013-03-08 03:00:05,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [618ec931] FINISH, GetHardwareInfoVDSCommand, log id: 6d0bb0a1 >2013-03-08 03:00:05,095 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-51) [671c0edf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:00:05,097 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-51) [1c2b0819] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:00:05,099 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-51) [1c2b0819] FINISH, ActivateVdsVDSCommand, log id: b3f152 >2013-03-08 03:00:05,118 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-51) [1c2b0819] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:00:05,119 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-51) [1c2b0819] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 03:00:05,121 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [1c2b0819] Autorecovering 1 storage domains >2013-03-08 03:00:05,121 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [1c2b0819] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 03:00:05,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-51) [57a7a64d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:00:05,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-51) [57a7a64d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:00 AM >2013-03-08 03:00:05,124 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-51) [57a7a64d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:00 AM >2013-03-08 03:00:05,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [693404a1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5f623392 >2013-03-08 03:00:05,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [693404a1] FINISH, GetHardwareInfoVDSCommand, log id: 5f623392 >2013-03-08 03:00:05,205 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-70) [71d28e52] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 03:00:05,208 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-70) [289fcd4a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 03:00:05,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4c7579f2 >2013-03-08 03:01:09,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 03:01:09,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:01:09,248 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 03:01:09,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 4c7579f2 >2013-03-08 03:01:09,252 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [16022bd8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:01:09,266 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [16022bd8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2fd3deec >2013-03-08 03:01:09,266 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [16022bd8] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 03:01:09,266 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [16022bd8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2fd3deec >2013-03-08 03:01:09,268 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [16022bd8] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 03:01:09,268 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [289fcd4a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 03:01:09,278 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-70) [28f91c1e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:01:09,280 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [28f91c1e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6457863d >2013-03-08 03:01:09,281 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [28f91c1e] FINISH, SetVdsStatusVDSCommand, log id: 6457863d >2013-03-08 03:01:09,287 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-70) [28f91c1e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 03:01:09,307 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-70) [105055ca] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:01:09,309 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-70) [621c65a2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:01:09,311 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-70) [621c65a2] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 03:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [ddd9e11] Autorecovering 1 hosts >2013-03-08 03:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [ddd9e11] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [26d4aa79] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [26d4aa79] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [26d4aa79] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4fe7584c >2013-03-08 03:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [26d4aa79] FINISH, SetVdsStatusVDSCommand, log id: 4fe7584c >2013-03-08 03:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-68) [26d4aa79] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 20510451 >2013-03-08 03:05:00,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [26d4aa79] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 47b49307 >2013-03-08 03:05:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [26d4aa79] FINISH, GetHardwareInfoVDSCommand, log id: 47b49307 >2013-03-08 03:05:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-68) [3a727475] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:05:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-68) [4c6a0cf4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:05:00,093 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-68) [4c6a0cf4] FINISH, ActivateVdsVDSCommand, log id: 20510451 >2013-03-08 03:05:00,112 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [4c6a0cf4] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:05:00,113 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-68) [4c6a0cf4] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 03:05:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [4c6a0cf4] Autorecovering 1 storage domains >2013-03-08 03:05:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [4c6a0cf4] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 03:05:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [79ed4924] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:05:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [79ed4924] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:05 AM >2013-03-08 03:05:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [79ed4924] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:05 AM >2013-03-08 03:05:00,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [206eb1f0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2078d2f2 >2013-03-08 03:05:00,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [206eb1f0] FINISH, GetHardwareInfoVDSCommand, log id: 2078d2f2 >2013-03-08 03:05:00,556 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-49) [1ad36c85] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 03:05:00,559 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-49) [78b3e006] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 03:05:00,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 66dfbf0c >2013-03-08 03:06:06,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 03:06:06,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:06:06,602 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 03:06:06,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 66dfbf0c >2013-03-08 03:06:06,606 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [24389322] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:06:06,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [24389322] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 60b4b387 >2013-03-08 03:06:06,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [24389322] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 03:06:06,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [24389322] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 60b4b387 >2013-03-08 03:06:06,622 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [24389322] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 03:06:06,622 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [78b3e006] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 03:06:06,632 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-49) [70046c94] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:06:06,634 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [70046c94] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5528144 >2013-03-08 03:06:06,635 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [70046c94] FINISH, SetVdsStatusVDSCommand, log id: 5528144 >2013-03-08 03:06:06,641 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-49) [70046c94] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 03:06:06,661 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-49) [67ca3f0a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:06:06,663 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-49) [c9ee581] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:06:06,664 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) [c9ee581] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 03:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [15f9bed0] Autorecovering 1 hosts >2013-03-08 03:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [15f9bed0] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [24931fc1] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [24931fc1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [24931fc1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6cfaf03b >2013-03-08 03:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [24931fc1] FINISH, SetVdsStatusVDSCommand, log id: 6cfaf03b >2013-03-08 03:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [24931fc1] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2d2e6011 >2013-03-08 03:10:00,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [24931fc1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3fa6b4ae >2013-03-08 03:10:00,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [24931fc1] FINISH, GetHardwareInfoVDSCommand, log id: 3fa6b4ae >2013-03-08 03:10:00,255 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [1647094e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:10:00,258 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [59f63ce4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:10:00,259 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [59f63ce4] FINISH, ActivateVdsVDSCommand, log id: 2d2e6011 >2013-03-08 03:10:00,279 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [59f63ce4] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:10:00,280 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [59f63ce4] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 03:10:00,281 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [59f63ce4] Autorecovering 1 storage domains >2013-03-08 03:10:00,281 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [59f63ce4] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 03:10:00,282 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [61dda42a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:10:00,282 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [61dda42a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:10 AM >2013-03-08 03:10:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [61dda42a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:10 AM >2013-03-08 03:10:00,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [52ed2d33] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2fcfb1e2 >2013-03-08 03:10:00,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [52ed2d33] FINISH, GetHardwareInfoVDSCommand, log id: 2fcfb1e2 >2013-03-08 03:10:00,909 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-4) [6ea897] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 03:10:00,912 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-4) [7bda4882] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 03:10:00,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7a84a75d >2013-03-08 03:11:03,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 03:11:03,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:11:03,915 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 03:11:03,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 7a84a75d >2013-03-08 03:11:03,919 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [20b48a41] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:11:03,934 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [20b48a41] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 13e957f >2013-03-08 03:11:03,934 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [20b48a41] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 03:11:03,934 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [20b48a41] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 13e957f >2013-03-08 03:11:03,936 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [20b48a41] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 03:11:03,936 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [7bda4882] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 03:11:03,946 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-4) [3ed5803e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:11:03,947 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [3ed5803e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 309a01a3 >2013-03-08 03:11:03,948 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [3ed5803e] FINISH, SetVdsStatusVDSCommand, log id: 309a01a3 >2013-03-08 03:11:03,955 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-4) [3ed5803e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 03:11:03,976 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-4) [2a68bcce] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:11:03,978 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-4) [75097004] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:11:03,979 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-4) [75097004] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 03:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [41d2a2b8] Autorecovering 1 hosts >2013-03-08 03:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [41d2a2b8] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-15) [6c5c0697] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-15) [6c5c0697] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [6c5c0697] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 70aa3f00 >2013-03-08 03:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [6c5c0697] FINISH, SetVdsStatusVDSCommand, log id: 70aa3f00 >2013-03-08 03:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-15) [6c5c0697] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 16e374b >2013-03-08 03:15:00,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [6c5c0697] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 358ec6cf >2013-03-08 03:15:00,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [6c5c0697] FINISH, GetHardwareInfoVDSCommand, log id: 358ec6cf >2013-03-08 03:15:00,420 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-15) [75e3012f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:15:00,422 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-15) [754b4020] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:15:00,423 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-15) [754b4020] FINISH, ActivateVdsVDSCommand, log id: 16e374b >2013-03-08 03:15:00,443 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-15) [754b4020] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:15:00,444 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-15) [754b4020] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 03:15:00,445 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [754b4020] Autorecovering 1 storage domains >2013-03-08 03:15:00,445 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [754b4020] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 03:15:00,446 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-15) [73c0a37b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:15:00,446 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-15) [73c0a37b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:15 AM >2013-03-08 03:15:00,449 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-15) [73c0a37b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:15 AM >2013-03-08 03:15:01,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [afe113b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 668dd71d >2013-03-08 03:15:01,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [afe113b] FINISH, GetHardwareInfoVDSCommand, log id: 668dd71d >2013-03-08 03:15:01,221 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-88) [179264ea] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 03:15:01,225 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-88) [4439d958] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 03:15:01,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5a89005f >2013-03-08 03:16:06,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 03:16:06,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:16:06,223 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 03:16:06,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 5a89005f >2013-03-08 03:16:06,226 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [324e108c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:16:06,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [324e108c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6472f54d >2013-03-08 03:16:06,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [324e108c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 03:16:06,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [324e108c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6472f54d >2013-03-08 03:16:06,243 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [324e108c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 03:16:06,244 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [4439d958] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 03:16:06,253 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-88) [5973cb3f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:16:06,255 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [5973cb3f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 59017fa3 >2013-03-08 03:16:06,256 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [5973cb3f] FINISH, SetVdsStatusVDSCommand, log id: 59017fa3 >2013-03-08 03:16:06,262 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-88) [5973cb3f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 03:16:06,282 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-88) [70082e26] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:16:06,284 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-88) [75f4dc2d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:16:06,286 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-88) [75f4dc2d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 03:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [727e03e5] Autorecovering 1 hosts >2013-03-08 03:20:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [727e03e5] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [652dab05] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [652dab05] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [652dab05] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6c0336a7 >2013-03-08 03:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [652dab05] FINISH, SetVdsStatusVDSCommand, log id: 6c0336a7 >2013-03-08 03:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [652dab05] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5793e3b7 >2013-03-08 03:20:00,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [652dab05] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 730f3a71 >2013-03-08 03:20:00,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [652dab05] FINISH, GetHardwareInfoVDSCommand, log id: 730f3a71 >2013-03-08 03:20:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-45) [3a29ca7f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:20:00,094 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-45) [1f9a9244] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:20:00,095 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [1f9a9244] FINISH, ActivateVdsVDSCommand, log id: 5793e3b7 >2013-03-08 03:20:00,114 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [1f9a9244] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:20:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [1f9a9244] Autorecovering 1 storage domains >2013-03-08 03:20:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [1f9a9244] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 03:20:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [5c619c0d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:20:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [5c619c0d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:20 AM >2013-03-08 03:20:00,122 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [5c619c0d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:20 AM >2013-03-08 03:20:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-6) START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3c11a123 >2013-03-08 03:20:00,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-6) FINISH, GetHardwareInfoVDSCommand, log id: 3c11a123 >2013-03-08 03:20:00,533 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-6) [69405f44] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 03:20:00,536 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-6) [51a09b3b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 03:20:00,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2e018139 >2013-03-08 03:21:04,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 03:21:04,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:21:04,526 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 03:21:04,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 2e018139 >2013-03-08 03:21:04,530 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [45c27bd7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:21:04,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [45c27bd7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 51e9c27c >2013-03-08 03:21:04,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [45c27bd7] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 03:21:04,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [45c27bd7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 51e9c27c >2013-03-08 03:21:04,546 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [45c27bd7] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 03:21:04,546 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [51a09b3b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 03:21:04,556 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-6) [4d4d43f2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:21:04,557 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-6) [4d4d43f2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 25e57f3e >2013-03-08 03:21:04,559 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-6) [4d4d43f2] FINISH, SetVdsStatusVDSCommand, log id: 25e57f3e >2013-03-08 03:21:04,565 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-6) [4d4d43f2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 03:21:04,585 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-6) [674921db] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:21:04,588 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-6) [7fa35030] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:21:04,590 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-6) [7fa35030] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 03:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [10f43d4e] Autorecovering 1 hosts >2013-03-08 03:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [10f43d4e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-86) [64edcfc9] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-86) [64edcfc9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [64edcfc9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1a9965dc >2013-03-08 03:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [64edcfc9] FINISH, SetVdsStatusVDSCommand, log id: 1a9965dc >2013-03-08 03:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-86) [64edcfc9] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 8b0747f >2013-03-08 03:25:00,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [64edcfc9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7958c35c >2013-03-08 03:25:00,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [64edcfc9] FINISH, GetHardwareInfoVDSCommand, log id: 7958c35c >2013-03-08 03:25:00,096 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-86) [59847a08] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:25:00,099 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-86) [49c86cec] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:25:00,100 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-86) [49c86cec] FINISH, ActivateVdsVDSCommand, log id: 8b0747f >2013-03-08 03:25:00,120 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-86) [49c86cec] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:25:00,121 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-86) [49c86cec] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 03:25:00,122 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [49c86cec] Autorecovering 1 storage domains >2013-03-08 03:25:00,122 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [49c86cec] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 03:25:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-86) [bef86da] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:25:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-86) [bef86da] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:25 AM >2013-03-08 03:25:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-86) [bef86da] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:25 AM >2013-03-08 03:25:01,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [1e8bb7c7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2a9ea640 >2013-03-08 03:25:01,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [1e8bb7c7] FINISH, GetHardwareInfoVDSCommand, log id: 2a9ea640 >2013-03-08 03:25:01,841 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-100) [5108a0cc] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 03:25:01,850 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-100) [248ec4d2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 03:25:01,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 207f3f97 >2013-03-08 03:26:06,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 03:26:06,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:26:06,893 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 03:26:06,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 207f3f97 >2013-03-08 03:26:06,897 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [352c56bd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:26:06,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [352c56bd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 53d925df >2013-03-08 03:26:06,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [352c56bd] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 03:26:06,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [352c56bd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 53d925df >2013-03-08 03:26:06,924 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [248ec4d2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 03:26:06,935 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-100) [6e59cb78] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:26:06,936 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [6e59cb78] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7d6f9faf >2013-03-08 03:26:06,937 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [6e59cb78] FINISH, SetVdsStatusVDSCommand, log id: 7d6f9faf >2013-03-08 03:26:06,945 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-100) [6e59cb78] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 03:26:06,966 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-100) [66e5ac2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:26:06,969 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-100) [74dcecd3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:26:06,971 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-100) [74dcecd3] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 03:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [bbb9476] Autorecovering 1 hosts >2013-03-08 03:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [bbb9476] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [2c18b325] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [2c18b325] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [2c18b325] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 33bf36d4 >2013-03-08 03:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [2c18b325] FINISH, SetVdsStatusVDSCommand, log id: 33bf36d4 >2013-03-08 03:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-75) [2c18b325] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6dfcce15 >2013-03-08 03:30:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [2c18b325] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 75f889ae >2013-03-08 03:30:00,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [2c18b325] FINISH, GetHardwareInfoVDSCommand, log id: 75f889ae >2013-03-08 03:30:00,218 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [ea235b3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:30:00,221 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [499db7dc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:30:00,222 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-75) [499db7dc] FINISH, ActivateVdsVDSCommand, log id: 6dfcce15 >2013-03-08 03:30:00,242 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [499db7dc] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:30:00,243 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-75) [499db7dc] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 03:30:00,244 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [499db7dc] Autorecovering 1 storage domains >2013-03-08 03:30:00,244 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [499db7dc] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 03:30:00,245 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [101fdc3f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:30:00,245 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [101fdc3f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:30 AM >2013-03-08 03:30:00,248 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [101fdc3f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:30 AM >2013-03-08 03:30:01,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [45fa64d6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7165654 >2013-03-08 03:30:01,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [45fa64d6] FINISH, GetHardwareInfoVDSCommand, log id: 7165654 >2013-03-08 03:30:01,209 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-41) [4afcd850] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 03:30:01,212 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-41) [598808d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 03:30:01,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3b875a09 >2013-03-08 03:30:33,722 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-30) [3fbbb1fc] Start refreshing all users data >2013-03-08 03:30:34,642 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) [151938f] Attempting to update VMs/Templates Ovf. >2013-03-08 03:30:34,643 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) [151938f] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 03:30:34,644 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) [151938f] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 03:30:34,644 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) [151938f] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 03:30:34,645 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) [151938f] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 03:30:34,645 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) [151938f] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 03:30:34,646 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-52) [151938f] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 03:31:05,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 03:31:05,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:31:05,214 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 03:31:05,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 3b875a09 >2013-03-08 03:31:05,218 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [45de7ed1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:31:05,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [45de7ed1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 53fd01a2 >2013-03-08 03:31:05,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [45de7ed1] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 03:31:05,232 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [45de7ed1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 53fd01a2 >2013-03-08 03:31:05,234 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [45de7ed1] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 03:31:05,234 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [598808d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 03:31:05,244 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-41) [7bc5843] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:31:05,245 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [7bc5843] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 38531371 >2013-03-08 03:31:05,246 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [7bc5843] FINISH, SetVdsStatusVDSCommand, log id: 38531371 >2013-03-08 03:31:05,253 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-41) [7bc5843] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 03:31:05,274 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-41) [72c9ed76] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:31:05,276 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-41) [7e50f0fe] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:31:05,277 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-41) [7e50f0fe] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 03:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [2a5d255b] Autorecovering 1 hosts >2013-03-08 03:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [2a5d255b] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-23) [49073f95] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-23) [49073f95] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-23) [49073f95] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 490ee099 >2013-03-08 03:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-23) [49073f95] FINISH, SetVdsStatusVDSCommand, log id: 490ee099 >2013-03-08 03:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-23) [49073f95] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1991460 >2013-03-08 03:35:00,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-23) [49073f95] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 69af6c37 >2013-03-08 03:35:00,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-23) [49073f95] FINISH, GetHardwareInfoVDSCommand, log id: 69af6c37 >2013-03-08 03:35:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-23) [73306f57] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:35:00,208 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-23) [50999ba] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:35:00,209 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-23) [50999ba] FINISH, ActivateVdsVDSCommand, log id: 1991460 >2013-03-08 03:35:00,228 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-23) [50999ba] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:35:00,229 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-23) [50999ba] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 03:35:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [50999ba] Autorecovering 1 storage domains >2013-03-08 03:35:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [50999ba] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 03:35:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-23) [4edfb6ce] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:35:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-23) [4edfb6ce] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:35 AM >2013-03-08 03:35:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-23) [4edfb6ce] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:35 AM >2013-03-08 03:35:02,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [bef86da] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 223ea8a5 >2013-03-08 03:35:02,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [bef86da] FINISH, GetHardwareInfoVDSCommand, log id: 223ea8a5 >2013-03-08 03:35:02,527 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-86) [30685005] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 03:35:02,530 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-86) [40ed34ca] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 03:35:02,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 766cd55e >2013-03-08 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (QuartzScheduler_Worker-65) [622b808a] AuditLogCleanupManager::deleteAgedOutAuditLogs - entered >2013-03-08 03:36:07,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 03:36:07,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:36:07,524 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 03:36:07,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 766cd55e >2013-03-08 03:36:07,528 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [565bcf60] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:36:07,542 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [565bcf60] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 88653fa >2013-03-08 03:36:07,542 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [565bcf60] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 03:36:07,542 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [565bcf60] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 88653fa >2013-03-08 03:36:07,544 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [565bcf60] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 03:36:07,544 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [40ed34ca] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 03:36:07,554 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-86) [6b01fb1b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:36:07,556 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [6b01fb1b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6ff7c0e4 >2013-03-08 03:36:07,557 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [6b01fb1b] FINISH, SetVdsStatusVDSCommand, log id: 6ff7c0e4 >2013-03-08 03:36:07,563 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-86) [6b01fb1b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 03:36:07,583 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-86) [7f208dfe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:36:07,585 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-86) [3245cecf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:36:07,587 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-86) [3245cecf] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 03:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-40) [7037220e] Autorecovering 1 hosts >2013-03-08 03:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-40) [7037220e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-40) [3dfe4bc8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-40) [3dfe4bc8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-40) [3dfe4bc8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 9120a69 >2013-03-08 03:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-40) [3dfe4bc8] FINISH, SetVdsStatusVDSCommand, log id: 9120a69 >2013-03-08 03:40:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-40) [3dfe4bc8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3a206e61 >2013-03-08 03:40:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-40) [3dfe4bc8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 758518cd >2013-03-08 03:40:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-40) [3dfe4bc8] FINISH, GetHardwareInfoVDSCommand, log id: 758518cd >2013-03-08 03:40:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-40) [41891779] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:40:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-40) [3aac55b5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:40:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-40) [3aac55b5] FINISH, ActivateVdsVDSCommand, log id: 3a206e61 >2013-03-08 03:40:00,221 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-40) [3aac55b5] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:40:00,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-40) [3aac55b5] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 03:40:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-40) [3aac55b5] Autorecovering 1 storage domains >2013-03-08 03:40:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-40) [3aac55b5] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 03:40:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-40) [37a27f22] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:40:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-40) [37a27f22] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:40 AM >2013-03-08 03:40:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-40) [37a27f22] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:40 AM >2013-03-08 03:40:01,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [3663092] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7d234561 >2013-03-08 03:40:01,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [3663092] FINISH, GetHardwareInfoVDSCommand, log id: 7d234561 >2013-03-08 03:40:01,833 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-37) [42865c80] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 03:40:01,836 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-37) [6f0feb5e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 03:40:01,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 39ab6f1b >2013-03-08 03:41:06,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 03:41:06,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:41:06,841 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 03:41:06,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 39ab6f1b >2013-03-08 03:41:06,845 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [2e88c25b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:41:06,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [2e88c25b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5f945425 >2013-03-08 03:41:06,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [2e88c25b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 03:41:06,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [2e88c25b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5f945425 >2013-03-08 03:41:06,867 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [2e88c25b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 03:41:06,867 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [6f0feb5e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 03:41:06,876 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-37) [23792a24] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:41:06,877 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [23792a24] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7a2d0c51 >2013-03-08 03:41:06,879 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [23792a24] FINISH, SetVdsStatusVDSCommand, log id: 7a2d0c51 >2013-03-08 03:41:06,884 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-37) [23792a24] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 03:41:06,904 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-37) [11e491f9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:41:06,906 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-37) [2aa04f43] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:41:06,908 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-37) [2aa04f43] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 03:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [2559972f] Autorecovering 1 hosts >2013-03-08 03:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [2559972f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [2e427bc7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [2e427bc7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [2e427bc7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 42952b8b >2013-03-08 03:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [2e427bc7] FINISH, SetVdsStatusVDSCommand, log id: 42952b8b >2013-03-08 03:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-27) [2e427bc7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 664dc500 >2013-03-08 03:45:00,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [2e427bc7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 29838047 >2013-03-08 03:45:00,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [2e427bc7] FINISH, GetHardwareInfoVDSCommand, log id: 29838047 >2013-03-08 03:45:00,274 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-27) [5400fc5f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:45:00,276 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-27) [2410994e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:45:00,277 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-27) [2410994e] FINISH, ActivateVdsVDSCommand, log id: 664dc500 >2013-03-08 03:45:00,297 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [2410994e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:45:00,298 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-27) [2410994e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 03:45:00,299 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [2410994e] Autorecovering 1 storage domains >2013-03-08 03:45:00,299 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [2410994e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 03:45:00,300 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [786de44] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:45:00,300 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [786de44] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:45 AM >2013-03-08 03:45:00,303 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [786de44] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:45 AM >2013-03-08 03:45:01,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [4a7d529f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 36b871e3 >2013-03-08 03:45:01,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [4a7d529f] FINISH, GetHardwareInfoVDSCommand, log id: 36b871e3 >2013-03-08 03:45:01,151 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-82) [7da9e500] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 03:45:01,154 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-82) [7dcd85d1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 03:45:01,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4749c280 >2013-03-08 03:46:07,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 03:46:07,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:46:07,425 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 03:46:07,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 4749c280 >2013-03-08 03:46:07,429 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [45f53839] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:46:07,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [45f53839] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4b40c347 >2013-03-08 03:46:07,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [45f53839] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 03:46:07,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [45f53839] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4b40c347 >2013-03-08 03:46:07,446 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [45f53839] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 03:46:07,446 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [7dcd85d1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 03:46:07,457 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-82) [65a1f620] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:46:07,458 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [65a1f620] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 15866ece >2013-03-08 03:46:07,460 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [65a1f620] FINISH, SetVdsStatusVDSCommand, log id: 15866ece >2013-03-08 03:46:07,465 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-82) [65a1f620] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 03:46:07,484 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-82) [44045648] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:46:07,492 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-82) [5f6cb773] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:46:07,494 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-82) [5f6cb773] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 03:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [43661eaa] Autorecovering 1 hosts >2013-03-08 03:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [43661eaa] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [1559339e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [1559339e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [1559339e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: b5950df >2013-03-08 03:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [1559339e] FINISH, SetVdsStatusVDSCommand, log id: b5950df >2013-03-08 03:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-78) [1559339e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7bbb491f >2013-03-08 03:50:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [1559339e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4ec9c8d8 >2013-03-08 03:50:00,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [1559339e] FINISH, GetHardwareInfoVDSCommand, log id: 4ec9c8d8 >2013-03-08 03:50:00,219 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-78) [535dc0ee] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:50:00,222 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-78) [38325f26] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:50:00,223 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-78) [38325f26] FINISH, ActivateVdsVDSCommand, log id: 7bbb491f >2013-03-08 03:50:00,242 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [38325f26] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:50:00,243 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-78) [38325f26] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 03:50:00,244 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [38325f26] Autorecovering 1 storage domains >2013-03-08 03:50:00,244 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [38325f26] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 03:50:00,245 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [31650cfa] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:50:00,245 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [31650cfa] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:50 AM >2013-03-08 03:50:00,248 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [31650cfa] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:50 AM >2013-03-08 03:50:01,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [13b6715d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 490b5736 >2013-03-08 03:50:01,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [13b6715d] FINISH, GetHardwareInfoVDSCommand, log id: 490b5736 >2013-03-08 03:50:01,749 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-96) [465b0172] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 03:50:01,753 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-96) [1e37178b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 03:50:01,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2425424e >2013-03-08 03:51:06,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 03:51:06,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:51:06,772 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 03:51:06,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 2425424e >2013-03-08 03:51:06,776 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [510123e4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:51:06,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [510123e4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4699459b >2013-03-08 03:51:06,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [510123e4] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 03:51:06,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [510123e4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4699459b >2013-03-08 03:51:06,792 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [510123e4] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 03:51:06,792 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [1e37178b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 03:51:06,802 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-96) [75fb0562] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:51:06,804 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [75fb0562] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3cbba80a >2013-03-08 03:51:06,805 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [75fb0562] FINISH, SetVdsStatusVDSCommand, log id: 3cbba80a >2013-03-08 03:51:06,811 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-96) [75fb0562] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 03:51:06,832 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-96) [4a1c1375] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:51:06,834 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-96) [70ea0b9a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:51:06,836 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-96) [70ea0b9a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 03:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [28be46c9] Autorecovering 1 hosts >2013-03-08 03:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [28be46c9] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [f08b708] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [f08b708] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [f08b708] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 288c1486 >2013-03-08 03:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [f08b708] FINISH, SetVdsStatusVDSCommand, log id: 288c1486 >2013-03-08 03:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-83) [f08b708] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6e0acfc3 >2013-03-08 03:55:00,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [f08b708] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 78e87ec8 >2013-03-08 03:55:00,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [f08b708] FINISH, GetHardwareInfoVDSCommand, log id: 78e87ec8 >2013-03-08 03:55:00,212 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-83) [75b52793] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:55:00,214 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-83) [78cf1872] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:55:00,215 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-83) [78cf1872] FINISH, ActivateVdsVDSCommand, log id: 6e0acfc3 >2013-03-08 03:55:00,234 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [78cf1872] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 03:55:00,235 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-83) [78cf1872] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 03:55:00,237 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [78cf1872] Autorecovering 1 storage domains >2013-03-08 03:55:00,237 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [78cf1872] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 03:55:00,237 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [433bdf74] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:55:00,238 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [433bdf74] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:55 AM >2013-03-08 03:55:00,240 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [433bdf74] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:55 AM >2013-03-08 03:55:01,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [f7c1140] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 386e26c6 >2013-03-08 03:55:01,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [f7c1140] FINISH, GetHardwareInfoVDSCommand, log id: 386e26c6 >2013-03-08 03:55:01,110 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-35) [282e25ea] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 03:55:01,113 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-35) [446f5489] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 03:55:01,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4b351c65 >2013-03-08 03:56:05,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 03:56:05,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 03:56:05,164 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 03:56:05,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 4b351c65 >2013-03-08 03:56:05,168 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [605c5ef8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 03:56:05,183 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [605c5ef8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 55e6de7e >2013-03-08 03:56:05,183 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [605c5ef8] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 03:56:05,183 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [605c5ef8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 55e6de7e >2013-03-08 03:56:05,185 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [605c5ef8] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 03:56:05,185 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [446f5489] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 03:56:05,195 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-35) [3f3b5841] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:56:05,196 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [3f3b5841] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 79eaaa5d >2013-03-08 03:56:05,197 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [3f3b5841] FINISH, SetVdsStatusVDSCommand, log id: 79eaaa5d >2013-03-08 03:56:05,204 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-35) [3f3b5841] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 03:56:05,243 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-35) [73315140] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:56:05,245 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-35) [1866c803] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 03:56:05,247 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-35) [1866c803] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 04:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [4ca19890] Autorecovering 1 hosts >2013-03-08 04:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [4ca19890] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [198f0244] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [198f0244] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [198f0244] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 61ce86f6 >2013-03-08 04:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [198f0244] FINISH, SetVdsStatusVDSCommand, log id: 61ce86f6 >2013-03-08 04:00:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-58) [198f0244] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 281331e0 >2013-03-08 04:00:00,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [198f0244] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 514161d1 >2013-03-08 04:00:00,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [198f0244] FINISH, GetHardwareInfoVDSCommand, log id: 514161d1 >2013-03-08 04:00:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-58) [3b9b6e20] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:00:00,208 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-58) [1bf75731] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:00:00,209 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-58) [1bf75731] FINISH, ActivateVdsVDSCommand, log id: 281331e0 >2013-03-08 04:00:00,228 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [1bf75731] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:00:00,229 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-58) [1bf75731] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 04:00:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [1bf75731] Autorecovering 1 storage domains >2013-03-08 04:00:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [1bf75731] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 04:00:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [550cdc8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:00:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [550cdc8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:00 AM >2013-03-08 04:00:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [550cdc8] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:00 AM >2013-03-08 04:00:02,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [92bbf84] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5b2927f >2013-03-08 04:00:02,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [92bbf84] FINISH, GetHardwareInfoVDSCommand, log id: 5b2927f >2013-03-08 04:00:02,495 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-64) [2695dd52] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 04:00:02,498 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-64) [61443cbf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 04:00:02,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 519bf4af >2013-03-08 04:01:08,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 04:01:08,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:01:08,552 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 04:01:08,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 519bf4af >2013-03-08 04:01:08,555 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [67222e6f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:01:08,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [67222e6f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6e9b8768 >2013-03-08 04:01:08,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [67222e6f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 04:01:08,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [67222e6f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6e9b8768 >2013-03-08 04:01:08,573 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [67222e6f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 04:01:08,574 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [61443cbf] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 04:01:08,584 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-64) [7818a393] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:01:08,585 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [7818a393] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 156786aa >2013-03-08 04:01:08,586 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [7818a393] FINISH, SetVdsStatusVDSCommand, log id: 156786aa >2013-03-08 04:01:08,592 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-64) [7818a393] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 04:01:08,612 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-64) [27b29a98] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:01:08,614 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-64) [a7eaa44] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:01:08,616 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-64) [a7eaa44] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 04:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [45069088] Autorecovering 1 hosts >2013-03-08 04:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [45069088] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-10) [2768cc50] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:05:00,010 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-10) [2768cc50] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:05:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [2768cc50] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6e166d6 >2013-03-08 04:05:00,019 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [2768cc50] FINISH, SetVdsStatusVDSCommand, log id: 6e166d6 >2013-03-08 04:05:00,022 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-10) [2768cc50] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 349e8325 >2013-03-08 04:05:00,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [2768cc50] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 75a964b >2013-03-08 04:05:00,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [2768cc50] FINISH, GetHardwareInfoVDSCommand, log id: 75a964b >2013-03-08 04:05:00,652 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-10) [1a27f4f3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:05:00,654 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-10) [2426fde6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:05:00,656 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-10) [2426fde6] FINISH, ActivateVdsVDSCommand, log id: 349e8325 >2013-03-08 04:05:00,675 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-10) [2426fde6] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:05:00,676 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-10) [2426fde6] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 04:05:00,677 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [2426fde6] Autorecovering 1 storage domains >2013-03-08 04:05:00,677 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [2426fde6] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 04:05:00,678 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-10) [4d8c6a6d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:05:00,678 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-10) [4d8c6a6d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:05 AM >2013-03-08 04:05:00,681 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-10) [4d8c6a6d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:05 AM >2013-03-08 04:05:02,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [394ce138] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3ea72a49 >2013-03-08 04:05:02,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [394ce138] FINISH, GetHardwareInfoVDSCommand, log id: 3ea72a49 >2013-03-08 04:05:02,863 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-19) [2366dd5a] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 04:05:02,867 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-19) [3d06df5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 04:05:02,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 749d0e96 >2013-03-08 04:06:08,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 04:06:08,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:06:08,875 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 04:06:08,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 749d0e96 >2013-03-08 04:06:08,879 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [7ee81fff] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:06:08,893 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [7ee81fff] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 534b5600 >2013-03-08 04:06:08,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [7ee81fff] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 04:06:08,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [7ee81fff] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 534b5600 >2013-03-08 04:06:08,896 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [7ee81fff] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 04:06:08,896 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [3d06df5] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 04:06:08,906 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-19) [537dd3a4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:06:08,907 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [537dd3a4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2ec50cb7 >2013-03-08 04:06:08,908 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [537dd3a4] FINISH, SetVdsStatusVDSCommand, log id: 2ec50cb7 >2013-03-08 04:06:08,915 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-19) [537dd3a4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 04:06:08,934 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-19) [7212ac6f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:06:08,937 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-19) [4895537] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:06:08,938 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-19) [4895537] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 04:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [1efa21cb] Autorecovering 1 hosts >2013-03-08 04:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [1efa21cb] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [6cf5a248] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [6cf5a248] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [6cf5a248] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6cb797d >2013-03-08 04:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [6cf5a248] FINISH, SetVdsStatusVDSCommand, log id: 6cb797d >2013-03-08 04:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [6cf5a248] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1d3af312 >2013-03-08 04:10:00,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [6cf5a248] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 52214016 >2013-03-08 04:10:00,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [6cf5a248] FINISH, GetHardwareInfoVDSCommand, log id: 52214016 >2013-03-08 04:10:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [16ce0383] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:10:00,095 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [3adf83f6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:10:00,096 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [3adf83f6] FINISH, ActivateVdsVDSCommand, log id: 1d3af312 >2013-03-08 04:10:00,116 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [3adf83f6] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:10:00,117 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [3adf83f6] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 04:10:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [3adf83f6] Autorecovering 1 storage domains >2013-03-08 04:10:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [3adf83f6] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 04:10:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [33d8d338] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:10:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [33d8d338] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:10 AM >2013-03-08 04:10:00,122 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [33d8d338] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:10 AM >2013-03-08 04:10:00,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [57dbec90] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3ed699a8 >2013-03-08 04:10:00,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [57dbec90] FINISH, GetHardwareInfoVDSCommand, log id: 3ed699a8 >2013-03-08 04:10:00,195 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-93) [782e07a] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 04:10:00,199 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-93) [5fdfe892] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 04:10:00,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 177c9bad >2013-03-08 04:11:04,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 04:11:04,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:11:04,206 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 04:11:04,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 177c9bad >2013-03-08 04:11:04,211 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [e316f64] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:11:04,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [e316f64] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 339087a6 >2013-03-08 04:11:04,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [e316f64] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 04:11:04,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [e316f64] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 339087a6 >2013-03-08 04:11:04,227 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [e316f64] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 04:11:04,228 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [5fdfe892] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 04:11:04,237 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-93) [7789bc9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:11:04,238 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [7789bc9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 797530e4 >2013-03-08 04:11:04,240 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [7789bc9] FINISH, SetVdsStatusVDSCommand, log id: 797530e4 >2013-03-08 04:11:04,246 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-93) [7789bc9] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 04:11:04,300 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [6a023923] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:11:04,310 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [3127cf5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:11:04,311 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-93) [3127cf5] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 04:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [57a7a64d] Autorecovering 1 hosts >2013-03-08 04:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [57a7a64d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:15:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-51) [ecca2be] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:15:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-51) [ecca2be] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [ecca2be] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7a0e5b4c >2013-03-08 04:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [ecca2be] FINISH, SetVdsStatusVDSCommand, log id: 7a0e5b4c >2013-03-08 04:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-51) [ecca2be] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 301bd992 >2013-03-08 04:15:00,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [ecca2be] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4abf293d >2013-03-08 04:15:00,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [ecca2be] FINISH, GetHardwareInfoVDSCommand, log id: 4abf293d >2013-03-08 04:15:00,234 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-51) [4f25aed9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:15:00,237 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-51) [4b5bb3d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:15:00,238 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-51) [4b5bb3d] FINISH, ActivateVdsVDSCommand, log id: 301bd992 >2013-03-08 04:15:00,257 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-51) [4b5bb3d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:15:00,258 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-51) [4b5bb3d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 04:15:00,259 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [4b5bb3d] Autorecovering 1 storage domains >2013-03-08 04:15:00,259 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [4b5bb3d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 04:15:00,260 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-51) [231273d2] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:15:00,260 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-51) [231273d2] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:15 AM >2013-03-08 04:15:00,263 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-51) [231273d2] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:15 AM >2013-03-08 04:15:01,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [1bd8a9ae] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1add67fe >2013-03-08 04:15:01,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [1bd8a9ae] FINISH, GetHardwareInfoVDSCommand, log id: 1add67fe >2013-03-08 04:15:01,554 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-8) [52e9625f] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 04:15:01,557 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-8) [7fb36e4c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 04:15:01,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1521639a >2013-03-08 04:16:05,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 04:16:05,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:16:05,551 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 04:16:05,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 1521639a >2013-03-08 04:16:05,555 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [44770663] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:16:05,569 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [44770663] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 492ed11e >2013-03-08 04:16:05,569 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [44770663] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 04:16:05,569 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [44770663] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 492ed11e >2013-03-08 04:16:05,571 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [44770663] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 04:16:05,571 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [7fb36e4c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 04:16:05,581 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-8) [50b3475d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:16:05,583 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [50b3475d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 41a09e67 >2013-03-08 04:16:05,584 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [50b3475d] FINISH, SetVdsStatusVDSCommand, log id: 41a09e67 >2013-03-08 04:16:05,591 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-8) [50b3475d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 04:16:05,610 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [5556be17] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:16:05,613 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [3884e2a8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:16:05,614 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-8) [3884e2a8] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 04:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [434b4905] Autorecovering 1 hosts >2013-03-08 04:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [434b4905] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:20:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [7411d81a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [7411d81a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [7411d81a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3bde005d >2013-03-08 04:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [7411d81a] FINISH, SetVdsStatusVDSCommand, log id: 3bde005d >2013-03-08 04:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [7411d81a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: dd54463 >2013-03-08 04:20:00,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [7411d81a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 220e2479 >2013-03-08 04:20:00,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [7411d81a] FINISH, GetHardwareInfoVDSCommand, log id: 220e2479 >2013-03-08 04:20:00,216 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [10b90813] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:20:00,218 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [5d1ffc0e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:20:00,219 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [5d1ffc0e] FINISH, ActivateVdsVDSCommand, log id: dd54463 >2013-03-08 04:20:00,238 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [5d1ffc0e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:20:00,239 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [5d1ffc0e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 04:20:00,241 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [5d1ffc0e] Autorecovering 1 storage domains >2013-03-08 04:20:00,241 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [5d1ffc0e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 04:20:00,241 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [61838c76] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:20:00,242 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [61838c76] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:20 AM >2013-03-08 04:20:00,244 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [61838c76] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:20 AM >2013-03-08 04:20:02,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [4febce43] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4cceaa5e >2013-03-08 04:20:02,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [4febce43] FINISH, GetHardwareInfoVDSCommand, log id: 4cceaa5e >2013-03-08 04:20:02,861 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-12) [c64ff8e] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 04:20:02,865 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-12) [25c31dbb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 04:20:02,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7750bc8b >2013-03-08 04:21:08,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 04:21:08,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:21:08,864 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 04:21:08,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 7750bc8b >2013-03-08 04:21:08,868 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [1c027ee] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:21:08,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1c027ee] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 39fee61f >2013-03-08 04:21:08,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [1c027ee] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 04:21:08,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1c027ee] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 39fee61f >2013-03-08 04:21:08,894 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [1c027ee] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 04:21:08,894 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [25c31dbb] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 04:21:08,903 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-12) [5bade355] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:21:08,904 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [5bade355] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 8f3a9db >2013-03-08 04:21:08,906 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [5bade355] FINISH, SetVdsStatusVDSCommand, log id: 8f3a9db >2013-03-08 04:21:08,912 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-12) [5bade355] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 04:21:08,931 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-12) [455a1d41] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:21:08,934 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-12) [2d106f9d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:21:08,935 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-12) [2d106f9d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 04:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [62df4fdf] Autorecovering 1 hosts >2013-03-08 04:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [62df4fdf] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [e5f6ad5] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [e5f6ad5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [e5f6ad5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6e80808e >2013-03-08 04:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [e5f6ad5] FINISH, SetVdsStatusVDSCommand, log id: 6e80808e >2013-03-08 04:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-20) [e5f6ad5] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 17809633 >2013-03-08 04:25:00,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [e5f6ad5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 66d6cae3 >2013-03-08 04:25:00,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [e5f6ad5] FINISH, GetHardwareInfoVDSCommand, log id: 66d6cae3 >2013-03-08 04:25:00,207 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-20) [30ec05e2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:25:00,209 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-20) [3c9d29e4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:25:00,211 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-20) [3c9d29e4] FINISH, ActivateVdsVDSCommand, log id: 17809633 >2013-03-08 04:25:00,230 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [3c9d29e4] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:25:00,231 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-20) [3c9d29e4] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 04:25:00,232 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [3c9d29e4] Autorecovering 1 storage domains >2013-03-08 04:25:00,232 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [3c9d29e4] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 04:25:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [3a756cd6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:25:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [3a756cd6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:25 AM >2013-03-08 04:25:00,236 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [3a756cd6] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:25 AM >2013-03-08 04:25:00,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [4f61cbc8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 306e8482 >2013-03-08 04:25:00,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [4f61cbc8] FINISH, GetHardwareInfoVDSCommand, log id: 306e8482 >2013-03-08 04:25:00,308 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-87) [580f70cb] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 04:25:00,311 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-87) [75adc141] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 04:25:00,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 693af349 >2013-03-08 04:26:05,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 04:26:05,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:26:05,346 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 04:26:05,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 693af349 >2013-03-08 04:26:05,349 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [56708a6b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:26:05,363 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [56708a6b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 57795a6 >2013-03-08 04:26:05,363 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [56708a6b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 04:26:05,363 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [56708a6b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 57795a6 >2013-03-08 04:26:05,365 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [56708a6b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 04:26:05,366 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [75adc141] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 04:26:05,375 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-87) [248741] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:26:05,377 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [248741] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 78fa9d28 >2013-03-08 04:26:05,378 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [248741] FINISH, SetVdsStatusVDSCommand, log id: 78fa9d28 >2013-03-08 04:26:05,384 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [248741] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 04:26:05,405 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [16de10e6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:26:05,407 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [53830029] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:26:05,408 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-87) [53830029] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 04:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [1d4b6bc2] Autorecovering 1 hosts >2013-03-08 04:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [1d4b6bc2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [688608df] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [688608df] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [688608df] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5a1513bc >2013-03-08 04:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [688608df] FINISH, SetVdsStatusVDSCommand, log id: 5a1513bc >2013-03-08 04:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-33) [688608df] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 34595221 >2013-03-08 04:30:00,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [688608df] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4395a885 >2013-03-08 04:30:00,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [688608df] FINISH, GetHardwareInfoVDSCommand, log id: 4395a885 >2013-03-08 04:30:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-33) [6ceea713] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:30:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-33) [78fffbfb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:30:00,206 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-33) [78fffbfb] FINISH, ActivateVdsVDSCommand, log id: 34595221 >2013-03-08 04:30:00,225 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [78fffbfb] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:30:00,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-33) [78fffbfb] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 04:30:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [78fffbfb] Autorecovering 1 storage domains >2013-03-08 04:30:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [78fffbfb] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 04:30:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [2353e7ee] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:30:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [2353e7ee] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:30 AM >2013-03-08 04:30:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [2353e7ee] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:30 AM >2013-03-08 04:30:02,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [79ed4924] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5c04a3f5 >2013-03-08 04:30:02,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [79ed4924] FINISH, GetHardwareInfoVDSCommand, log id: 5c04a3f5 >2013-03-08 04:30:02,661 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-68) [1c7118a1] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 04:30:02,664 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-68) [328034fa] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 04:30:02,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 24af765b >2013-03-08 04:30:33,723 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-42) [2e0e0b36] Start refreshing all users data >2013-03-08 04:30:34,646 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-35) [1866c803] Attempting to update VMs/Templates Ovf. >2013-03-08 04:30:34,647 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-35) [1866c803] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 04:30:34,648 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-35) [1866c803] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 04:30:34,648 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-35) [1866c803] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 04:30:34,649 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-35) [1866c803] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 04:30:34,649 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-35) [1866c803] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 04:30:34,649 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-35) [1866c803] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 04:31:06,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 04:31:06,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:31:06,716 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 04:31:06,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 24af765b >2013-03-08 04:31:06,720 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [21c956c7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:31:06,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [21c956c7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2fd49407 >2013-03-08 04:31:06,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [21c956c7] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 04:31:06,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [21c956c7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2fd49407 >2013-03-08 04:31:06,739 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [328034fa] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 04:31:06,750 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-68) [270b05c7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:31:06,751 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [270b05c7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2208eb41 >2013-03-08 04:31:06,752 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [270b05c7] FINISH, SetVdsStatusVDSCommand, log id: 2208eb41 >2013-03-08 04:31:06,759 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-68) [270b05c7] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 04:31:06,780 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-68) [2f0b3930] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:31:06,782 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-68) [69c17889] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:31:06,784 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-68) [69c17889] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 04:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [2b30cfaa] Autorecovering 1 hosts >2013-03-08 04:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [2b30cfaa] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-56) [387baff2] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-56) [387baff2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [387baff2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 56cba3fe >2013-03-08 04:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [387baff2] FINISH, SetVdsStatusVDSCommand, log id: 56cba3fe >2013-03-08 04:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-56) [387baff2] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4d15cce9 >2013-03-08 04:35:00,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [387baff2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5ef66132 >2013-03-08 04:35:00,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [387baff2] FINISH, GetHardwareInfoVDSCommand, log id: 5ef66132 >2013-03-08 04:35:00,245 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-56) [63b66b92] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:35:00,248 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-56) [676f8d04] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:35:00,249 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-56) [676f8d04] FINISH, ActivateVdsVDSCommand, log id: 4d15cce9 >2013-03-08 04:35:00,268 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-56) [676f8d04] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:35:00,269 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-56) [676f8d04] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 04:35:00,270 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [676f8d04] Autorecovering 1 storage domains >2013-03-08 04:35:00,271 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [676f8d04] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 04:35:00,271 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-56) [1299f4e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:35:00,271 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-56) [1299f4e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:35 AM >2013-03-08 04:35:00,274 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-56) [1299f4e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:35 AM >2013-03-08 04:35:00,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [657cd278] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: ce63e31 >2013-03-08 04:35:00,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [657cd278] FINISH, GetHardwareInfoVDSCommand, log id: ce63e31 >2013-03-08 04:35:01,022 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-61) [2f15826a] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 04:35:01,025 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-61) [166a2945] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 04:35:01,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 65fad482 >2013-03-08 04:36:04,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 04:36:04,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:36:04,063 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 04:36:04,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 65fad482 >2013-03-08 04:36:04,067 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [4fbeb53] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:36:04,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [4fbeb53] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2e83b157 >2013-03-08 04:36:04,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [4fbeb53] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 04:36:04,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [4fbeb53] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2e83b157 >2013-03-08 04:36:04,083 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [4fbeb53] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 04:36:04,083 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [166a2945] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 04:36:04,095 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-61) [11ee61f5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:36:04,097 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [11ee61f5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3d98d0a8 >2013-03-08 04:36:04,098 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [11ee61f5] FINISH, SetVdsStatusVDSCommand, log id: 3d98d0a8 >2013-03-08 04:36:04,105 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-61) [11ee61f5] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 04:36:04,134 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-61) [4f575e9b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:36:04,137 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-61) [43b479b3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:36:04,138 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-61) [43b479b3] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 04:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [484afdc7] Autorecovering 1 hosts >2013-03-08 04:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [484afdc7] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [25a98f1b] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [25a98f1b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [25a98f1b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3e76a268 >2013-03-08 04:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [25a98f1b] FINISH, SetVdsStatusVDSCommand, log id: 3e76a268 >2013-03-08 04:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [25a98f1b] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3629a0d4 >2013-03-08 04:40:00,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [25a98f1b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 819d311 >2013-03-08 04:40:00,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [25a98f1b] FINISH, GetHardwareInfoVDSCommand, log id: 819d311 >2013-03-08 04:40:00,188 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [555a1225] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:40:00,190 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [69b7f7df] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:40:00,191 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [69b7f7df] FINISH, ActivateVdsVDSCommand, log id: 3629a0d4 >2013-03-08 04:40:00,211 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [69b7f7df] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:40:00,211 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [69b7f7df] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 04:40:00,213 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [69b7f7df] Autorecovering 1 storage domains >2013-03-08 04:40:00,213 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [69b7f7df] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 04:40:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [bc54636] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:40:00,214 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [bc54636] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:40 AM >2013-03-08 04:40:00,216 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [bc54636] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:40 AM >2013-03-08 04:40:01,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [434cb3a8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7ef1124e >2013-03-08 04:40:01,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [434cb3a8] FINISH, GetHardwareInfoVDSCommand, log id: 7ef1124e >2013-03-08 04:40:01,385 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-5) [4c89fc4f] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 04:40:01,388 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-5) [362fbb54] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 04:40:01,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 66304dac >2013-03-08 04:41:05,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 04:41:05,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:41:05,409 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 04:41:05,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 66304dac >2013-03-08 04:41:05,413 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [176830ea] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:41:05,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [176830ea] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c9bbab4 >2013-03-08 04:41:05,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [176830ea] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 04:41:05,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [176830ea] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c9bbab4 >2013-03-08 04:41:05,430 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [176830ea] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 04:41:05,431 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [362fbb54] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 04:41:05,440 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-5) [1198d56a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:41:05,441 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [1198d56a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4ed1730e >2013-03-08 04:41:05,443 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [1198d56a] FINISH, SetVdsStatusVDSCommand, log id: 4ed1730e >2013-03-08 04:41:05,449 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-5) [1198d56a] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 04:41:05,471 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-5) [1791923f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:41:05,474 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-5) [75ba1457] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:41:05,475 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-5) [75ba1457] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 04:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [2b03ae05] Autorecovering 1 hosts >2013-03-08 04:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [2b03ae05] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [5efa7d3b] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [5efa7d3b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [5efa7d3b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3ebce0ac >2013-03-08 04:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [5efa7d3b] FINISH, SetVdsStatusVDSCommand, log id: 3ebce0ac >2013-03-08 04:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-21) [5efa7d3b] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 99eb6e6 >2013-03-08 04:45:00,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [5efa7d3b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3d4e1a7e >2013-03-08 04:45:00,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [5efa7d3b] FINISH, GetHardwareInfoVDSCommand, log id: 3d4e1a7e >2013-03-08 04:45:00,463 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-21) [46858d6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:45:00,465 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-21) [4bba4cdf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:45:00,467 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-21) [4bba4cdf] FINISH, ActivateVdsVDSCommand, log id: 99eb6e6 >2013-03-08 04:45:00,486 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [4bba4cdf] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:45:00,487 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-21) [4bba4cdf] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 04:45:00,488 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [4bba4cdf] Autorecovering 1 storage domains >2013-03-08 04:45:00,488 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [4bba4cdf] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 04:45:00,489 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [6fbec909] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:45:00,489 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [6fbec909] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:45 AM >2013-03-08 04:45:00,492 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [6fbec909] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:45 AM >2013-03-08 04:45:02,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [dbdac2d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 32658783 >2013-03-08 04:45:02,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [dbdac2d] FINISH, GetHardwareInfoVDSCommand, log id: 32658783 >2013-03-08 04:45:02,718 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-57) [566c68ff] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 04:45:02,721 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-57) [3769ba55] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 04:45:02,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 64c0b161 >2013-03-08 04:46:08,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 04:46:08,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:46:08,775 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 04:46:08,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 64c0b161 >2013-03-08 04:46:08,779 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [62b3fca8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:46:08,794 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [62b3fca8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 76f8befb >2013-03-08 04:46:08,794 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [62b3fca8] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 04:46:08,794 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [62b3fca8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 76f8befb >2013-03-08 04:46:08,796 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [62b3fca8] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 04:46:08,796 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [3769ba55] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 04:46:08,806 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-57) [4f08d657] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:46:08,808 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [4f08d657] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3b9695 >2013-03-08 04:46:08,809 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [4f08d657] FINISH, SetVdsStatusVDSCommand, log id: 3b9695 >2013-03-08 04:46:08,815 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-57) [4f08d657] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 04:46:08,836 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-57) [6062b867] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:46:08,838 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-57) [438879d2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:46:08,840 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-57) [438879d2] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 04:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [70ea0b9a] Autorecovering 1 hosts >2013-03-08 04:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [70ea0b9a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:50:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [6c3ca65f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [6c3ca65f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:50:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [6c3ca65f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3839177f >2013-03-08 04:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [6c3ca65f] FINISH, SetVdsStatusVDSCommand, log id: 3839177f >2013-03-08 04:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [6c3ca65f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 75cd33d >2013-03-08 04:50:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [6c3ca65f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 43fbb859 >2013-03-08 04:50:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [6c3ca65f] FINISH, GetHardwareInfoVDSCommand, log id: 43fbb859 >2013-03-08 04:50:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-96) [6f10681f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:50:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-96) [c933057] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:50:00,093 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [c933057] FINISH, ActivateVdsVDSCommand, log id: 75cd33d >2013-03-08 04:50:00,112 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [c933057] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:50:00,113 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-96) [c933057] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 04:50:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [c933057] Autorecovering 1 storage domains >2013-03-08 04:50:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [c933057] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 04:50:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [a99f831] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:50:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [a99f831] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:50 AM >2013-03-08 04:50:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [a99f831] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:50 AM >2013-03-08 04:50:03,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [44b5caad] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 479e851b >2013-03-08 04:50:03,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [44b5caad] FINISH, GetHardwareInfoVDSCommand, log id: 479e851b >2013-03-08 04:50:03,088 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-66) [2fa35bc0] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 04:50:03,091 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-66) [115dbc5e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 04:50:03,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 408069fc >2013-03-08 04:51:08,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 04:51:08,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:51:08,114 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 04:51:08,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 408069fc >2013-03-08 04:51:08,118 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [5ff426c3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:51:08,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [5ff426c3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1e815ad0 >2013-03-08 04:51:08,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [5ff426c3] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 04:51:08,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [5ff426c3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1e815ad0 >2013-03-08 04:51:08,135 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [5ff426c3] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 04:51:08,135 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [115dbc5e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 04:51:08,146 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-66) [791b7d90] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:51:08,147 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [791b7d90] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6e7af887 >2013-03-08 04:51:08,148 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [791b7d90] FINISH, SetVdsStatusVDSCommand, log id: 6e7af887 >2013-03-08 04:51:08,155 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-66) [791b7d90] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 04:51:08,174 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-66) [4338bf22] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:51:08,177 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-66) [6da2ca63] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:51:08,178 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-66) [6da2ca63] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 04:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [38bfed06] Autorecovering 1 hosts >2013-03-08 04:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [38bfed06] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [265138d0] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [265138d0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [265138d0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7adbe4f >2013-03-08 04:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [265138d0] FINISH, SetVdsStatusVDSCommand, log id: 7adbe4f >2013-03-08 04:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-44) [265138d0] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: b67021a >2013-03-08 04:55:00,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [265138d0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 549633b1 >2013-03-08 04:55:00,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [265138d0] FINISH, GetHardwareInfoVDSCommand, log id: 549633b1 >2013-03-08 04:55:00,098 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-44) [197bd794] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:55:00,101 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-44) [17ad443c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:55:00,102 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-44) [17ad443c] FINISH, ActivateVdsVDSCommand, log id: b67021a >2013-03-08 04:55:00,122 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [17ad443c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 04:55:00,123 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-44) [17ad443c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 04:55:00,124 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [17ad443c] Autorecovering 1 storage domains >2013-03-08 04:55:00,124 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [17ad443c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 04:55:00,125 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [5f361390] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:55:00,125 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [5f361390] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:55 AM >2013-03-08 04:55:00,127 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [5f361390] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:55 AM >2013-03-08 04:55:02,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [5c619c0d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 28aba293 >2013-03-08 04:55:02,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [5c619c0d] FINISH, GetHardwareInfoVDSCommand, log id: 28aba293 >2013-03-08 04:55:02,426 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-45) [15bf5b16] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 04:55:02,430 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-45) [5133f809] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 04:55:02,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 265980f1 >2013-03-08 04:56:06,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 04:56:06,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 04:56:06,468 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 04:56:06,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 265980f1 >2013-03-08 04:56:06,471 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [46efe759] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 04:56:06,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [46efe759] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7820d117 >2013-03-08 04:56:06,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [46efe759] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 04:56:06,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [46efe759] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7820d117 >2013-03-08 04:56:06,488 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [46efe759] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 04:56:06,488 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [5133f809] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 04:56:06,498 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-45) [1212b443] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:56:06,499 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [1212b443] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4b79fb20 >2013-03-08 04:56:06,501 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [1212b443] FINISH, SetVdsStatusVDSCommand, log id: 4b79fb20 >2013-03-08 04:56:06,507 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-45) [1212b443] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 04:56:06,527 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-45) [12eadf7e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:56:06,530 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-45) [31135343] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 04:56:06,531 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-45) [31135343] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 05:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [34620c93] Autorecovering 1 hosts >2013-03-08 05:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [34620c93] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-28) [459657d4] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-28) [459657d4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [459657d4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7c4623c4 >2013-03-08 05:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [459657d4] FINISH, SetVdsStatusVDSCommand, log id: 7c4623c4 >2013-03-08 05:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-28) [459657d4] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3aeafc15 >2013-03-08 05:00:00,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [459657d4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2070d977 >2013-03-08 05:00:00,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [459657d4] FINISH, GetHardwareInfoVDSCommand, log id: 2070d977 >2013-03-08 05:00:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-28) [6055649] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:00:00,095 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-28) [7aa0f8f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:00:00,096 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-28) [7aa0f8f] FINISH, ActivateVdsVDSCommand, log id: 3aeafc15 >2013-03-08 05:00:00,116 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-28) [7aa0f8f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:00:00,117 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-28) [7aa0f8f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 05:00:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [7aa0f8f] Autorecovering 1 storage domains >2013-03-08 05:00:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [7aa0f8f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 05:00:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-28) [59e33a68] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:00:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-28) [59e33a68] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 5:00 AM >2013-03-08 05:00:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-28) [59e33a68] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 5:00 AM >2013-03-08 05:00:00,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-71) [3640068] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5f45ac2c >2013-03-08 05:00:00,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-71) [3640068] FINISH, GetHardwareInfoVDSCommand, log id: 5f45ac2c >2013-03-08 05:00:00,776 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-71) [1230008c] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 05:00:00,780 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-71) [37c1127e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 05:00:00,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3c7d7329 >2013-03-08 05:01:06,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 05:01:06,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:01:06,776 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 05:01:06,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 3c7d7329 >2013-03-08 05:01:06,780 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [38e82120] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:01:06,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [38e82120] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4959e2b >2013-03-08 05:01:06,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [38e82120] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 05:01:06,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [38e82120] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4959e2b >2013-03-08 05:01:06,802 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [38e82120] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 05:01:06,802 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [37c1127e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 05:01:06,813 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-71) [32c5eae2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:01:06,814 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-71) [32c5eae2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2826fd8d >2013-03-08 05:01:06,815 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-71) [32c5eae2] FINISH, SetVdsStatusVDSCommand, log id: 2826fd8d >2013-03-08 05:01:06,821 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-71) [32c5eae2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 05:01:06,841 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-71) [17cb3573] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:01:06,843 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-71) [34890b99] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:01:06,845 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-71) [34890b99] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 05:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [d093265] Autorecovering 1 hosts >2013-03-08 05:05:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [d093265] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-3) [280f93ef] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-3) [280f93ef] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-3) [280f93ef] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 52bba393 >2013-03-08 05:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-3) [280f93ef] FINISH, SetVdsStatusVDSCommand, log id: 52bba393 >2013-03-08 05:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-3) [280f93ef] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 63b8cea2 >2013-03-08 05:05:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-3) [280f93ef] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3f476bb8 >2013-03-08 05:05:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-3) [280f93ef] FINISH, GetHardwareInfoVDSCommand, log id: 3f476bb8 >2013-03-08 05:05:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-3) [3563c174] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:05:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-3) [1f666341] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:05:00,094 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-3) [1f666341] FINISH, ActivateVdsVDSCommand, log id: 63b8cea2 >2013-03-08 05:05:00,113 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-3) [1f666341] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:05:00,114 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) [1f666341] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 05:05:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [1f666341] Autorecovering 1 storage domains >2013-03-08 05:05:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [1f666341] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 05:05:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-3) [445f1240] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:05:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-3) [445f1240] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 5:05 AM >2013-03-08 05:05:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-3) [445f1240] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 5:05 AM >2013-03-08 05:05:01,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-40) [37a27f22] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 128aa9a0 >2013-03-08 05:05:01,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-40) [37a27f22] FINISH, GetHardwareInfoVDSCommand, log id: 128aa9a0 >2013-03-08 05:05:01,085 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-40) [275d697f] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 05:05:01,088 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-40) [61a5df0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 05:05:01,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 46d16c72 >2013-03-08 05:06:05,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 05:06:05,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:06:05,100 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 05:06:05,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 46d16c72 >2013-03-08 05:06:05,104 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [53d7d0e8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:06:05,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [53d7d0e8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: e5c0a65 >2013-03-08 05:06:05,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [53d7d0e8] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 05:06:05,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [53d7d0e8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: e5c0a65 >2013-03-08 05:06:05,138 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [53d7d0e8] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 05:06:05,138 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [61a5df0] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 05:06:05,148 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-40) [b3ec810] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:06:05,150 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-40) [b3ec810] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6f611dbb >2013-03-08 05:06:05,151 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-40) [b3ec810] FINISH, SetVdsStatusVDSCommand, log id: 6f611dbb >2013-03-08 05:06:05,159 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-40) [b3ec810] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 05:06:05,180 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-40) [4c149a72] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:06:05,183 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-40) [327c1650] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:06:05,184 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-40) [327c1650] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 05:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [158c1d87] Autorecovering 1 hosts >2013-03-08 05:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [158c1d87] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [2638f3f0] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [2638f3f0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [2638f3f0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 69d87195 >2013-03-08 05:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [2638f3f0] FINISH, SetVdsStatusVDSCommand, log id: 69d87195 >2013-03-08 05:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-90) [2638f3f0] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6d08123 >2013-03-08 05:10:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [2638f3f0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6aef884d >2013-03-08 05:10:00,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [2638f3f0] FINISH, GetHardwareInfoVDSCommand, log id: 6aef884d >2013-03-08 05:10:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-90) [47df0d05] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:10:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-90) [3cbbaa78] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:10:00,094 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-90) [3cbbaa78] FINISH, ActivateVdsVDSCommand, log id: 6d08123 >2013-03-08 05:10:00,113 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [3cbbaa78] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:10:00,114 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [3cbbaa78] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 05:10:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [3cbbaa78] Autorecovering 1 storage domains >2013-03-08 05:10:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [3cbbaa78] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 05:10:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [36b909f5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:10:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [36b909f5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 5:10 AM >2013-03-08 05:10:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [36b909f5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 5:10 AM >2013-03-08 05:10:02,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [786de44] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 17bacdcb >2013-03-08 05:10:02,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [786de44] FINISH, GetHardwareInfoVDSCommand, log id: 17bacdcb >2013-03-08 05:10:02,432 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-27) [5d5abe5f] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 05:10:02,436 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-27) [d67695a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 05:10:02,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 349e165e >2013-03-08 05:11:07,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 05:11:07,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:11:07,442 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 05:11:07,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 349e165e >2013-03-08 05:11:07,446 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [41a75305] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:11:07,460 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [41a75305] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 60cfa43c >2013-03-08 05:11:07,460 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [41a75305] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 05:11:07,460 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [41a75305] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 60cfa43c >2013-03-08 05:11:07,462 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [41a75305] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 05:11:07,463 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [d67695a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 05:11:07,473 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-27) [613356f7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:11:07,474 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [613356f7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2b6ed02e >2013-03-08 05:11:07,475 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [613356f7] FINISH, SetVdsStatusVDSCommand, log id: 2b6ed02e >2013-03-08 05:11:07,482 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-27) [613356f7] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 05:11:07,503 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-27) [273cacb6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:11:07,505 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-27) [6f75aaff] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:11:07,507 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-27) [6f75aaff] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 05:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [31650cfa] Autorecovering 1 hosts >2013-03-08 05:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [31650cfa] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [7d6b28fb] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [7d6b28fb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [7d6b28fb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 49a05b11 >2013-03-08 05:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [7d6b28fb] FINISH, SetVdsStatusVDSCommand, log id: 49a05b11 >2013-03-08 05:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-78) [7d6b28fb] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 65a6e2ff >2013-03-08 05:15:00,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [7d6b28fb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6e1db6e7 >2013-03-08 05:15:00,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [7d6b28fb] FINISH, GetHardwareInfoVDSCommand, log id: 6e1db6e7 >2013-03-08 05:15:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-78) [23bc7e47] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:15:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-78) [2f313c1b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:15:00,205 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-78) [2f313c1b] FINISH, ActivateVdsVDSCommand, log id: 65a6e2ff >2013-03-08 05:15:00,225 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [2f313c1b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:15:00,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-78) [2f313c1b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 05:15:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [2f313c1b] Autorecovering 1 storage domains >2013-03-08 05:15:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [2f313c1b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 05:15:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [547a2bbf] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:15:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [547a2bbf] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 5:15 AM >2013-03-08 05:15:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [547a2bbf] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 5:15 AM >2013-03-08 05:15:01,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [39799c8b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1e07143f >2013-03-08 05:15:01,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [39799c8b] FINISH, GetHardwareInfoVDSCommand, log id: 1e07143f >2013-03-08 05:15:01,760 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-69) [24e2ca24] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 05:15:01,764 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-69) [791bd00e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 05:15:01,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 38426ebf >2013-03-08 05:16:08,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 05:16:08,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:16:08,059 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 05:16:08,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 38426ebf >2013-03-08 05:16:08,063 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [5507a14b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:16:08,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [5507a14b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 43a88f49 >2013-03-08 05:16:08,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [5507a14b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 05:16:08,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [5507a14b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 43a88f49 >2013-03-08 05:16:08,079 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [5507a14b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 05:16:08,080 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [791bd00e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 05:16:08,089 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-69) [76d4206c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:16:08,091 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [76d4206c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4456164f >2013-03-08 05:16:08,092 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [76d4206c] FINISH, SetVdsStatusVDSCommand, log id: 4456164f >2013-03-08 05:16:08,098 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-69) [76d4206c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 05:16:08,119 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-69) [1732beea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:16:08,121 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-69) [1c2fa1f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:16:08,123 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-69) [1c2fa1f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 05:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [151938f] Autorecovering 1 hosts >2013-03-08 05:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [151938f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [2c8434fd] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [2c8434fd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [2c8434fd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6bc75848 >2013-03-08 05:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [2c8434fd] FINISH, SetVdsStatusVDSCommand, log id: 6bc75848 >2013-03-08 05:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [2c8434fd] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7b0017fc >2013-03-08 05:20:00,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [2c8434fd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 63e21597 >2013-03-08 05:20:00,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [2c8434fd] FINISH, GetHardwareInfoVDSCommand, log id: 63e21597 >2013-03-08 05:20:00,431 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [3f35a4aa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:20:00,433 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [9de9c8d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:20:00,434 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [9de9c8d] FINISH, ActivateVdsVDSCommand, log id: 7b0017fc >2013-03-08 05:20:00,454 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [9de9c8d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:20:00,454 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-52) [9de9c8d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 05:20:00,456 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [9de9c8d] Autorecovering 1 storage domains >2013-03-08 05:20:00,456 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [9de9c8d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 05:20:00,456 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [47f2e6ff] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:20:00,457 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [47f2e6ff] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 5:20 AM >2013-03-08 05:20:00,460 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [47f2e6ff] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 5:20 AM >2013-03-08 05:20:02,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [7e3606b9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6a7de99f >2013-03-08 05:20:02,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [7e3606b9] FINISH, GetHardwareInfoVDSCommand, log id: 6a7de99f >2013-03-08 05:20:02,355 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-38) [200abae8] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 05:20:02,358 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-38) [2a6e86ef] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 05:20:02,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4cce2b2 >2013-03-08 05:21:06,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 05:21:06,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:21:06,385 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 05:21:06,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 4cce2b2 >2013-03-08 05:21:06,389 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [703c151d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:21:06,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [703c151d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5f5a4d02 >2013-03-08 05:21:06,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [703c151d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 05:21:06,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [703c151d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5f5a4d02 >2013-03-08 05:21:06,405 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [703c151d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 05:21:06,406 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [2a6e86ef] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 05:21:06,415 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-38) [3bb2edaa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:21:06,416 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [3bb2edaa] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 20633c2 >2013-03-08 05:21:06,417 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [3bb2edaa] FINISH, SetVdsStatusVDSCommand, log id: 20633c2 >2013-03-08 05:21:06,424 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-38) [3bb2edaa] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 05:21:06,443 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-38) [5a0788cb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:21:06,446 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-38) [71b9d5d9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:21:06,447 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-38) [71b9d5d9] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 05:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [622b808a] Autorecovering 1 hosts >2013-03-08 05:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [622b808a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [54dc5252] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [54dc5252] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [54dc5252] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1af553c >2013-03-08 05:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [54dc5252] FINISH, SetVdsStatusVDSCommand, log id: 1af553c >2013-03-08 05:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-65) [54dc5252] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 18e101ca >2013-03-08 05:25:00,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [54dc5252] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 48641bd3 >2013-03-08 05:25:00,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [54dc5252] FINISH, GetHardwareInfoVDSCommand, log id: 48641bd3 >2013-03-08 05:25:00,212 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-65) [34c475b3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:25:00,214 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-65) [d1088af] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:25:00,215 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-65) [d1088af] FINISH, ActivateVdsVDSCommand, log id: 18e101ca >2013-03-08 05:25:00,235 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [d1088af] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:25:00,236 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-65) [d1088af] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 05:25:00,237 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [d1088af] Autorecovering 1 storage domains >2013-03-08 05:25:00,237 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [d1088af] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 05:25:00,238 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [7692caee] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:25:00,238 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [7692caee] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 5:25 AM >2013-03-08 05:25:00,240 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [7692caee] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 5:25 AM >2013-03-08 05:25:00,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [2ab636cb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5aa53569 >2013-03-08 05:25:00,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [2ab636cb] FINISH, GetHardwareInfoVDSCommand, log id: 5aa53569 >2013-03-08 05:25:00,698 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-31) [4fea0a29] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 05:25:00,702 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-31) [4a182d5a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 05:25:00,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2ebb8d1e >2013-03-08 05:26:04,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 05:26:04,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:26:04,738 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 05:26:04,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 2ebb8d1e >2013-03-08 05:26:04,742 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [2b754dc0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:26:04,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [2b754dc0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 319d4855 >2013-03-08 05:26:04,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [2b754dc0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 05:26:04,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [2b754dc0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 319d4855 >2013-03-08 05:26:04,758 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [2b754dc0] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 05:26:04,758 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [4a182d5a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 05:26:04,768 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-31) [528231c2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:26:04,769 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [528231c2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6c54216c >2013-03-08 05:26:04,770 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [528231c2] FINISH, SetVdsStatusVDSCommand, log id: 6c54216c >2013-03-08 05:26:04,776 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-31) [528231c2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 05:26:04,801 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-31) [7ad7d9c9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:26:04,804 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-31) [33ae847e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:26:04,805 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-31) [33ae847e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 05:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [6fe3b474] Autorecovering 1 hosts >2013-03-08 05:30:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [6fe3b474] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-39) [4585771d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-39) [4585771d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [4585771d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: de8bf19 >2013-03-08 05:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [4585771d] FINISH, SetVdsStatusVDSCommand, log id: de8bf19 >2013-03-08 05:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-39) [4585771d] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 75fe7fb9 >2013-03-08 05:30:00,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [4585771d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 50b8e14 >2013-03-08 05:30:00,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [4585771d] FINISH, GetHardwareInfoVDSCommand, log id: 50b8e14 >2013-03-08 05:30:00,296 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-39) [1caa9ad0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:30:00,299 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-39) [1252e256] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:30:00,300 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-39) [1252e256] FINISH, ActivateVdsVDSCommand, log id: 75fe7fb9 >2013-03-08 05:30:00,319 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-39) [1252e256] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:30:00,320 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-39) [1252e256] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 05:30:00,321 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [1252e256] Autorecovering 1 storage domains >2013-03-08 05:30:00,322 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [1252e256] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 05:30:00,322 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-39) [2e56b86b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:30:00,322 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-39) [2e56b86b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 5:30 AM >2013-03-08 05:30:00,325 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-39) [2e56b86b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 5:30 AM >2013-03-08 05:30:01,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [69c17889] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 69e37dbf >2013-03-08 05:30:02,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [69c17889] FINISH, GetHardwareInfoVDSCommand, log id: 69e37dbf >2013-03-08 05:30:02,059 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-68) [1b0ed586] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 05:30:02,063 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-68) [560322c1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 05:30:02,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5561d61a >2013-03-08 05:30:33,725 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-15) [73c0a37b] Start refreshing all users data >2013-03-08 05:30:34,649 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-33) [2353e7ee] Attempting to update VMs/Templates Ovf. >2013-03-08 05:30:34,650 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-33) [2353e7ee] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 05:30:34,651 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-33) [2353e7ee] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 05:30:34,651 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-33) [2353e7ee] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 05:30:34,652 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-33) [2353e7ee] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 05:30:34,652 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-33) [2353e7ee] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 05:30:34,652 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-33) [2353e7ee] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 05:31:07,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 05:31:07,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:31:07,083 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 05:31:07,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 5561d61a >2013-03-08 05:31:07,087 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [6eeb8f6d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:31:07,101 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [6eeb8f6d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 58e30c70 >2013-03-08 05:31:07,101 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [6eeb8f6d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 05:31:07,101 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [6eeb8f6d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 58e30c70 >2013-03-08 05:31:07,106 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [560322c1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 05:31:07,116 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-68) [551d0454] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:31:07,117 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [551d0454] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 580d21a0 >2013-03-08 05:31:07,118 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [551d0454] FINISH, SetVdsStatusVDSCommand, log id: 580d21a0 >2013-03-08 05:31:07,125 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-68) [551d0454] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 05:31:07,146 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-68) [7012d14f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:31:07,149 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-68) [24d08e02] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:31:07,150 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-68) [24d08e02] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 05:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [715fa4a2] Autorecovering 1 hosts >2013-03-08 05:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [715fa4a2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [65433c87] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [65433c87] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [65433c87] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 43255efc >2013-03-08 05:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [65433c87] FINISH, SetVdsStatusVDSCommand, log id: 43255efc >2013-03-08 05:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [65433c87] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4e22b275 >2013-03-08 05:35:00,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [65433c87] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 36f8cf3e >2013-03-08 05:35:00,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [65433c87] FINISH, GetHardwareInfoVDSCommand, log id: 36f8cf3e >2013-03-08 05:35:00,309 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [35efe0f8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:35:00,312 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [2a06b603] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:35:00,313 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [2a06b603] FINISH, ActivateVdsVDSCommand, log id: 4e22b275 >2013-03-08 05:35:00,332 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [2a06b603] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:35:00,333 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [2a06b603] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 05:35:00,335 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [2a06b603] Autorecovering 1 storage domains >2013-03-08 05:35:00,335 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [2a06b603] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 05:35:00,335 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [15343e77] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:35:00,335 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [15343e77] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 5:35 AM >2013-03-08 05:35:00,338 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [15343e77] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 5:35 AM >2013-03-08 05:35:01,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-85) [504e04de] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4cb0295c >2013-03-08 05:35:01,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-85) [504e04de] FINISH, GetHardwareInfoVDSCommand, log id: 4cb0295c >2013-03-08 05:35:01,399 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-85) [5604e4c8] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 05:35:01,402 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-85) [4f6b5ac4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 05:35:01,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: d18ce0d >2013-03-08 05:36:07,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 05:36:07,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:36:07,694 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 05:36:07,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: d18ce0d >2013-03-08 05:36:07,697 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [68abd9a4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:36:07,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [68abd9a4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 72d8f18a >2013-03-08 05:36:07,711 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [68abd9a4] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 05:36:07,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [68abd9a4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 72d8f18a >2013-03-08 05:36:07,714 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [68abd9a4] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 05:36:07,714 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [4f6b5ac4] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 05:36:07,724 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-85) [3c6752c0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:36:07,726 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-85) [3c6752c0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 57adfb37 >2013-03-08 05:36:07,727 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-85) [3c6752c0] FINISH, SetVdsStatusVDSCommand, log id: 57adfb37 >2013-03-08 05:36:07,733 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-85) [3c6752c0] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 05:36:07,753 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-85) [35bc7c5b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:36:07,756 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-85) [6853da68] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:36:07,757 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-85) [6853da68] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 05:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [70dd4adf] Autorecovering 1 hosts >2013-03-08 05:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [70dd4adf] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [64e82f14] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [64e82f14] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [64e82f14] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 508685d7 >2013-03-08 05:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [64e82f14] FINISH, SetVdsStatusVDSCommand, log id: 508685d7 >2013-03-08 05:40:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-2) [64e82f14] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 58e2d3e8 >2013-03-08 05:40:00,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [64e82f14] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1ff36f78 >2013-03-08 05:40:00,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [64e82f14] FINISH, GetHardwareInfoVDSCommand, log id: 1ff36f78 >2013-03-08 05:40:00,210 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [e8d5052] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:40:00,212 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [34d23164] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:40:00,214 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-2) [34d23164] FINISH, ActivateVdsVDSCommand, log id: 58e2d3e8 >2013-03-08 05:40:00,233 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [34d23164] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:40:00,234 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [34d23164] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 05:40:00,235 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [34d23164] Autorecovering 1 storage domains >2013-03-08 05:40:00,235 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [34d23164] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 05:40:00,236 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [8598a6c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:40:00,236 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [8598a6c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 5:40 AM >2013-03-08 05:40:00,239 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [8598a6c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 5:40 AM >2013-03-08 05:40:01,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [2e56b86b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1fcf8f99 >2013-03-08 05:40:01,961 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [2e56b86b] FINISH, GetHardwareInfoVDSCommand, log id: 1fcf8f99 >2013-03-08 05:40:01,999 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-39) [62a27414] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 05:40:02,002 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-39) [629c9d78] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 05:40:02,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3d938106 >2013-03-08 05:41:05,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 05:41:05,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:41:05,987 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 05:41:05,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 3d938106 >2013-03-08 05:41:05,990 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [243a7c15] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:41:06,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [243a7c15] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4fb9fe7d >2013-03-08 05:41:06,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [243a7c15] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 05:41:06,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [243a7c15] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4fb9fe7d >2013-03-08 05:41:06,006 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [243a7c15] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 05:41:06,007 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [629c9d78] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 05:41:06,016 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-39) [3d969d30] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:41:06,018 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [3d969d30] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 459606ad >2013-03-08 05:41:06,019 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [3d969d30] FINISH, SetVdsStatusVDSCommand, log id: 459606ad >2013-03-08 05:41:06,025 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-39) [3d969d30] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 05:41:06,046 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-39) [793d1a73] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:41:06,049 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-39) [47f066f5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:41:06,050 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-39) [47f066f5] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 05:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [16d2016b] Autorecovering 1 hosts >2013-03-08 05:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [16d2016b] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-76) [757bf407] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-76) [757bf407] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [757bf407] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: fd5877 >2013-03-08 05:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [757bf407] FINISH, SetVdsStatusVDSCommand, log id: fd5877 >2013-03-08 05:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-76) [757bf407] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 31164c7c >2013-03-08 05:45:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [757bf407] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 176a1927 >2013-03-08 05:45:00,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [757bf407] FINISH, GetHardwareInfoVDSCommand, log id: 176a1927 >2013-03-08 05:45:00,224 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-76) [3698b9b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:45:00,228 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-76) [7d31bc92] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:45:00,230 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-76) [7d31bc92] FINISH, ActivateVdsVDSCommand, log id: 31164c7c >2013-03-08 05:45:00,263 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-76) [7d31bc92] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:45:00,264 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-76) [7d31bc92] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 05:45:00,266 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [7d31bc92] Autorecovering 1 storage domains >2013-03-08 05:45:00,266 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [7d31bc92] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 05:45:00,266 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-76) [6b04b768] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:45:00,266 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-76) [6b04b768] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 5:45 AM >2013-03-08 05:45:00,275 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-76) [6b04b768] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 5:45 AM >2013-03-08 05:45:00,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [33d8d338] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3f933ba6 >2013-03-08 05:45:00,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [33d8d338] FINISH, GetHardwareInfoVDSCommand, log id: 3f933ba6 >2013-03-08 05:45:00,353 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-26) [7260c1d6] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 05:45:00,357 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-26) [2df8587b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 05:45:00,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5bd5954 >2013-03-08 05:46:05,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 05:46:05,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:46:05,343 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 05:46:05,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 5bd5954 >2013-03-08 05:46:05,346 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [7433f0cb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:46:05,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [7433f0cb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 62aa1098 >2013-03-08 05:46:05,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [7433f0cb] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 05:46:05,361 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [7433f0cb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 62aa1098 >2013-03-08 05:46:05,363 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [7433f0cb] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 05:46:05,363 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [2df8587b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 05:46:05,373 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-26) [2cb675d2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:46:05,375 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [2cb675d2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 18c6e2ab >2013-03-08 05:46:05,376 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [2cb675d2] FINISH, SetVdsStatusVDSCommand, log id: 18c6e2ab >2013-03-08 05:46:05,382 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [2cb675d2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 05:46:05,402 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [5caa712d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:46:05,404 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [6ab4d6b6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:46:05,405 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-26) [6ab4d6b6] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 05:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [5c7fd00e] Autorecovering 1 hosts >2013-03-08 05:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [5c7fd00e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [7a32ccb0] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [7a32ccb0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [7a32ccb0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5367ce93 >2013-03-08 05:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [7a32ccb0] FINISH, SetVdsStatusVDSCommand, log id: 5367ce93 >2013-03-08 05:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [7a32ccb0] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 10ee2c >2013-03-08 05:50:01,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [7a32ccb0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 250cd583 >2013-03-08 05:50:01,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [7a32ccb0] FINISH, GetHardwareInfoVDSCommand, log id: 250cd583 >2013-03-08 05:50:01,066 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-63) [47571d8f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:50:01,068 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-63) [3ecfe576] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:50:01,070 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [3ecfe576] FINISH, ActivateVdsVDSCommand, log id: 10ee2c >2013-03-08 05:50:01,089 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [3ecfe576] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:50:01,090 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-63) [3ecfe576] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 05:50:01,091 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [3ecfe576] Autorecovering 1 storage domains >2013-03-08 05:50:01,091 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [3ecfe576] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 05:50:01,092 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [128fb420] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:50:01,092 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [128fb420] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 5:50 AM >2013-03-08 05:50:01,095 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [128fb420] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 5:50 AM >2013-03-08 05:50:02,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [75f4dc2d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 52472052 >2013-03-08 05:50:02,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [75f4dc2d] FINISH, GetHardwareInfoVDSCommand, log id: 52472052 >2013-03-08 05:50:02,652 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-88) [ad737f6] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 05:50:02,655 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-88) [17b81790] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 05:50:02,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 61d64d14 >2013-03-08 05:51:08,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 05:51:08,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:51:08,670 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 05:51:08,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 61d64d14 >2013-03-08 05:51:08,675 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [5d8ac7f6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:51:08,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [5d8ac7f6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 152e6260 >2013-03-08 05:51:08,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [5d8ac7f6] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 05:51:08,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [5d8ac7f6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 152e6260 >2013-03-08 05:51:08,691 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [5d8ac7f6] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 05:51:08,691 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [17b81790] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 05:51:08,701 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-88) [4918258d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:51:08,703 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [4918258d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 26999fa6 >2013-03-08 05:51:08,704 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [4918258d] FINISH, SetVdsStatusVDSCommand, log id: 26999fa6 >2013-03-08 05:51:08,711 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-88) [4918258d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 05:51:08,730 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-88) [7eec16a6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:51:08,733 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-88) [448102d1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:51:08,734 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-88) [448102d1] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 05:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [7e50f0fe] Autorecovering 1 hosts >2013-03-08 05:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [7e50f0fe] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [ec71979] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [ec71979] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [ec71979] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 384252e4 >2013-03-08 05:55:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [ec71979] FINISH, SetVdsStatusVDSCommand, log id: 384252e4 >2013-03-08 05:55:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-41) [ec71979] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 494e324 >2013-03-08 05:55:00,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [ec71979] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 67cfa32c >2013-03-08 05:55:00,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [ec71979] FINISH, GetHardwareInfoVDSCommand, log id: 67cfa32c >2013-03-08 05:55:00,094 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-41) [6f336b23] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:55:00,096 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-41) [12356e29] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:55:00,098 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-41) [12356e29] FINISH, ActivateVdsVDSCommand, log id: 494e324 >2013-03-08 05:55:00,117 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [12356e29] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 05:55:00,118 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-41) [12356e29] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 05:55:00,120 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [12356e29] Autorecovering 1 storage domains >2013-03-08 05:55:00,120 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [12356e29] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 05:55:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [316d0cb8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:55:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [316d0cb8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 5:55 AM >2013-03-08 05:55:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [316d0cb8] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 5:55 AM >2013-03-08 05:55:02,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [61838c76] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 34f9c652 >2013-03-08 05:55:02,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [61838c76] FINISH, GetHardwareInfoVDSCommand, log id: 34f9c652 >2013-03-08 05:55:02,979 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-43) [4b2c5503] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 05:55:02,983 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-43) [341c62aa] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 05:55:02,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 41581a8f >2013-03-08 05:56:07,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 05:56:07,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 05:56:07,019 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 05:56:07,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 41581a8f >2013-03-08 05:56:07,023 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [567f4bf6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 05:56:07,038 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [567f4bf6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 150d6531 >2013-03-08 05:56:07,038 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [567f4bf6] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 05:56:07,038 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [567f4bf6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 150d6531 >2013-03-08 05:56:07,040 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [567f4bf6] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 05:56:07,040 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [341c62aa] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 05:56:07,056 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-43) [1dd85b1a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:56:07,057 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [1dd85b1a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 547193c3 >2013-03-08 05:56:07,059 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [1dd85b1a] FINISH, SetVdsStatusVDSCommand, log id: 547193c3 >2013-03-08 05:56:07,065 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [1dd85b1a] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 05:56:07,086 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [2051865f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:56:07,089 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [2ac2a35] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 05:56:07,090 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-43) [2ac2a35] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 06:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [61dda42a] Autorecovering 1 hosts >2013-03-08 06:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [61dda42a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [5fba811c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [5fba811c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [5fba811c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5bbed7b8 >2013-03-08 06:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [5fba811c] FINISH, SetVdsStatusVDSCommand, log id: 5bbed7b8 >2013-03-08 06:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [5fba811c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 9375837 >2013-03-08 06:00:00,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [5fba811c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 11aa6e06 >2013-03-08 06:00:00,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [5fba811c] FINISH, GetHardwareInfoVDSCommand, log id: 11aa6e06 >2013-03-08 06:00:00,242 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [827ea8d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:00:00,244 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [14b15054] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:00:00,246 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [14b15054] FINISH, ActivateVdsVDSCommand, log id: 9375837 >2013-03-08 06:00:00,265 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [14b15054] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:00:00,266 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [14b15054] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 06:00:00,267 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [14b15054] Autorecovering 1 storage domains >2013-03-08 06:00:00,267 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [14b15054] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 06:00:00,268 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [533b0fff] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:00:00,268 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [533b0fff] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 6:00 AM >2013-03-08 06:00:00,271 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [533b0fff] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 6:00 AM >2013-03-08 06:00:01,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [1299f4e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4665cdff >2013-03-08 06:00:01,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [1299f4e] FINISH, GetHardwareInfoVDSCommand, log id: 4665cdff >2013-03-08 06:00:01,330 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-56) [18ff9b0e] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 06:00:01,333 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-56) [4b59ba72] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 06:00:01,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7da3cb7f >2013-03-08 06:01:07,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 06:01:07,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:01:07,349 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 06:01:07,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 7da3cb7f >2013-03-08 06:01:07,352 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [6eb6cd23] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:01:07,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6eb6cd23] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 29b23e4d >2013-03-08 06:01:07,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [6eb6cd23] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 06:01:07,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6eb6cd23] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 29b23e4d >2013-03-08 06:01:07,368 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [6eb6cd23] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 06:01:07,369 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [4b59ba72] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 06:01:07,379 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-56) [7c90f870] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:01:07,380 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [7c90f870] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 243a0ba5 >2013-03-08 06:01:07,381 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [7c90f870] FINISH, SetVdsStatusVDSCommand, log id: 243a0ba5 >2013-03-08 06:01:07,388 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-56) [7c90f870] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 06:01:07,412 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-56) [b11d83d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:01:07,415 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-56) [d1e2a77] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:01:07,417 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-56) [d1e2a77] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 06:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [bc54636] Autorecovering 1 hosts >2013-03-08 06:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [bc54636] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [229d0ce8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [229d0ce8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [229d0ce8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7cf1bcbd >2013-03-08 06:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [229d0ce8] FINISH, SetVdsStatusVDSCommand, log id: 7cf1bcbd >2013-03-08 06:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [229d0ce8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 731fcd53 >2013-03-08 06:05:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [229d0ce8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 13e20172 >2013-03-08 06:05:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [229d0ce8] FINISH, GetHardwareInfoVDSCommand, log id: 13e20172 >2013-03-08 06:05:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [49c0c8d8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:05:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [63a743a4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:05:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [63a743a4] FINISH, ActivateVdsVDSCommand, log id: 731fcd53 >2013-03-08 06:05:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [63a743a4] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:05:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [63a743a4] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 06:05:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [63a743a4] Autorecovering 1 storage domains >2013-03-08 06:05:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [63a743a4] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 06:05:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [68cfb3b9] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:05:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [68cfb3b9] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 6:05 AM >2013-03-08 06:05:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [68cfb3b9] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 6:05 AM >2013-03-08 06:05:01,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [f65bd94] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 64f376a2 >2013-03-08 06:05:01,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [f65bd94] FINISH, GetHardwareInfoVDSCommand, log id: 64f376a2 >2013-03-08 06:05:01,665 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-99) [576fad10] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 06:05:01,668 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-99) [4a1a84dc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 06:05:01,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6471b1e7 >2013-03-08 06:06:06,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 06:06:06,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:06:06,660 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 06:06:06,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 6471b1e7 >2013-03-08 06:06:06,664 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [7c008575] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:06:06,678 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [7c008575] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3b39e539 >2013-03-08 06:06:06,678 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [7c008575] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 06:06:06,678 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [7c008575] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3b39e539 >2013-03-08 06:06:06,680 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [7c008575] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 06:06:06,681 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [4a1a84dc] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 06:06:06,691 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-99) [76521417] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:06:06,692 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [76521417] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3d8728fc >2013-03-08 06:06:06,693 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [76521417] FINISH, SetVdsStatusVDSCommand, log id: 3d8728fc >2013-03-08 06:06:06,700 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-99) [76521417] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 06:06:06,732 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-99) [1b416aa5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:06:06,735 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-99) [409ef71f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:06:06,736 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-99) [409ef71f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 06:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [2e8c04d6] Autorecovering 1 hosts >2013-03-08 06:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [2e8c04d6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [48c71951] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [48c71951] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [48c71951] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 43069ed5 >2013-03-08 06:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [48c71951] FINISH, SetVdsStatusVDSCommand, log id: 43069ed5 >2013-03-08 06:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [48c71951] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1607bcc3 >2013-03-08 06:10:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [48c71951] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 281181af >2013-03-08 06:10:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [48c71951] FINISH, GetHardwareInfoVDSCommand, log id: 281181af >2013-03-08 06:10:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-16) [7897b338] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:10:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-16) [754c7b60] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:10:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [754c7b60] FINISH, ActivateVdsVDSCommand, log id: 1607bcc3 >2013-03-08 06:10:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [754c7b60] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:10:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-16) [754c7b60] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 06:10:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [754c7b60] Autorecovering 1 storage domains >2013-03-08 06:10:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [754c7b60] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 06:10:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [2616e83f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:10:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [2616e83f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 6:10 AM >2013-03-08 06:10:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [2616e83f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 6:10 AM >2013-03-08 06:10:00,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [34358fb3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1bfc0fc5 >2013-03-08 06:10:00,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [34358fb3] FINISH, GetHardwareInfoVDSCommand, log id: 1bfc0fc5 >2013-03-08 06:10:00,986 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-53) [b7aefe2] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 06:10:00,989 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-53) [74579d87] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 06:10:00,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 56160bc5 >2013-03-08 06:11:04,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 06:11:04,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:11:04,991 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 06:11:04,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 56160bc5 >2013-03-08 06:11:04,994 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [19154d52] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:11:05,009 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [19154d52] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1402872f >2013-03-08 06:11:05,009 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [19154d52] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 06:11:05,009 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [19154d52] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1402872f >2013-03-08 06:11:05,011 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [19154d52] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 06:11:05,011 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [74579d87] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 06:11:05,021 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-53) [2ffafe7c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:11:05,022 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [2ffafe7c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2077836b >2013-03-08 06:11:05,024 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [2ffafe7c] FINISH, SetVdsStatusVDSCommand, log id: 2077836b >2013-03-08 06:11:05,029 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-53) [2ffafe7c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 06:11:05,050 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-53) [7047409e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:11:05,052 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-53) [46a42c3f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:11:05,054 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-53) [46a42c3f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 06:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) Autorecovering 1 hosts >2013-03-08 06:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-60) [1da48d7e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-60) [1da48d7e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [1da48d7e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 340fcccf >2013-03-08 06:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [1da48d7e] FINISH, SetVdsStatusVDSCommand, log id: 340fcccf >2013-03-08 06:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-60) [1da48d7e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1fb90367 >2013-03-08 06:15:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [1da48d7e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5afcd129 >2013-03-08 06:15:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [1da48d7e] FINISH, GetHardwareInfoVDSCommand, log id: 5afcd129 >2013-03-08 06:15:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-60) [318e4b8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:15:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-60) [606f5f5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:15:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-60) [606f5f5] FINISH, ActivateVdsVDSCommand, log id: 1fb90367 >2013-03-08 06:15:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-60) [606f5f5] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:15:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-60) [606f5f5] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 06:15:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [606f5f5] Autorecovering 1 storage domains >2013-03-08 06:15:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [606f5f5] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 06:15:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-60) [20224cd5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:15:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-60) [20224cd5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 6:15 AM >2013-03-08 06:15:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-60) [20224cd5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 6:15 AM >2013-03-08 06:15:02,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-98) [2369fac0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 642f2ff0 >2013-03-08 06:15:02,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-98) [2369fac0] FINISH, GetHardwareInfoVDSCommand, log id: 642f2ff0 >2013-03-08 06:15:02,302 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-98) [3921550e] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 06:15:02,305 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-98) [2e610091] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 06:15:02,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 35e118fe >2013-03-08 06:16:07,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 06:16:07,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:16:07,284 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 06:16:07,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 35e118fe >2013-03-08 06:16:07,288 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [1f0253ac] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:16:07,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1f0253ac] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7434631d >2013-03-08 06:16:07,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [1f0253ac] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 06:16:07,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1f0253ac] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7434631d >2013-03-08 06:16:07,310 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [1f0253ac] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 06:16:07,311 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [2e610091] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 06:16:07,321 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-98) [2e997202] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:16:07,323 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-98) [2e997202] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7e82b1ee >2013-03-08 06:16:07,324 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-98) [2e997202] FINISH, SetVdsStatusVDSCommand, log id: 7e82b1ee >2013-03-08 06:16:07,330 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-98) [2e997202] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 06:16:07,349 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-98) [663b0899] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:16:07,352 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-98) [2ce40ea1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:16:07,353 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-98) [2ce40ea1] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 06:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [1866c803] Autorecovering 1 hosts >2013-03-08 06:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [1866c803] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [3ab5965a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [3ab5965a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [3ab5965a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7d5a6e70 >2013-03-08 06:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [3ab5965a] FINISH, SetVdsStatusVDSCommand, log id: 7d5a6e70 >2013-03-08 06:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-35) [3ab5965a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 33e4ffd6 >2013-03-08 06:20:00,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [3ab5965a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 20c1b59f >2013-03-08 06:20:00,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [3ab5965a] FINISH, GetHardwareInfoVDSCommand, log id: 20c1b59f >2013-03-08 06:20:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-35) [7a708aa4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:20:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-35) [552ee52c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:20:00,205 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-35) [552ee52c] FINISH, ActivateVdsVDSCommand, log id: 33e4ffd6 >2013-03-08 06:20:00,224 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [552ee52c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:20:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [552ee52c] Autorecovering 1 storage domains >2013-03-08 06:20:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [552ee52c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 06:20:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [465df3e5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:20:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [465df3e5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 6:20 AM >2013-03-08 06:20:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [465df3e5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 6:20 AM >2013-03-08 06:20:01,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [71b9d5d9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 61b1586f >2013-03-08 06:20:01,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [71b9d5d9] FINISH, GetHardwareInfoVDSCommand, log id: 61b1586f >2013-03-08 06:20:01,595 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-38) [3e8de53] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 06:20:01,599 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-38) [4a59477] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 06:20:01,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2ccaf431 >2013-03-08 06:21:06,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 06:21:06,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:21:06,652 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 06:21:06,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 2ccaf431 >2013-03-08 06:21:06,656 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [57908326] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:21:06,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [57908326] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 612e8304 >2013-03-08 06:21:06,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [57908326] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 06:21:06,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [57908326] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 612e8304 >2013-03-08 06:21:06,672 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [57908326] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 06:21:06,673 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [4a59477] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 06:21:06,682 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-38) [1bba5926] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:21:06,684 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [1bba5926] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3a79b241 >2013-03-08 06:21:06,685 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [1bba5926] FINISH, SetVdsStatusVDSCommand, log id: 3a79b241 >2013-03-08 06:21:06,690 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-38) [1bba5926] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 06:21:06,710 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-38) [86719f2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:21:06,712 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-38) [1c68a073] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:21:06,714 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-38) [1c68a073] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 06:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [7daa393a] Autorecovering 1 hosts >2013-03-08 06:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [7daa393a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-25) [409bc585] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-25) [409bc585] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [409bc585] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 132a8020 >2013-03-08 06:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [409bc585] FINISH, SetVdsStatusVDSCommand, log id: 132a8020 >2013-03-08 06:25:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-25) [409bc585] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 76176248 >2013-03-08 06:25:00,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [409bc585] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1f69b5b3 >2013-03-08 06:25:00,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [409bc585] FINISH, GetHardwareInfoVDSCommand, log id: 1f69b5b3 >2013-03-08 06:25:00,534 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-25) [570ae28f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:25:00,536 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-25) [24fce46] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:25:00,538 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-25) [24fce46] FINISH, ActivateVdsVDSCommand, log id: 76176248 >2013-03-08 06:25:00,557 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-25) [24fce46] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:25:00,558 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-25) [24fce46] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 06:25:00,559 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [24fce46] Autorecovering 1 storage domains >2013-03-08 06:25:00,559 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [24fce46] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 06:25:00,560 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-25) [4a818f99] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:25:00,560 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-25) [4a818f99] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 6:25 AM >2013-03-08 06:25:00,563 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-25) [4a818f99] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 6:25 AM >2013-03-08 06:25:00,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [59e33a68] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2ca49aaf >2013-03-08 06:25:00,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [59e33a68] FINISH, GetHardwareInfoVDSCommand, log id: 2ca49aaf >2013-03-08 06:25:00,956 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-28) [5916e0c8] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 06:25:00,960 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-28) [5c48c1f4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 06:25:00,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6ac0c32f >2013-03-08 06:26:06,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 06:26:06,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:26:06,016 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 06:26:06,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 6ac0c32f >2013-03-08 06:26:06,020 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [400dca1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:26:06,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [400dca1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7073220f >2013-03-08 06:26:06,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [400dca1] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 06:26:06,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [400dca1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7073220f >2013-03-08 06:26:06,036 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [400dca1] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 06:26:06,036 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [5c48c1f4] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 06:26:06,057 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-28) [5ad337f1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:26:06,059 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [5ad337f1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7af9e4ad >2013-03-08 06:26:06,060 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [5ad337f1] FINISH, SetVdsStatusVDSCommand, log id: 7af9e4ad >2013-03-08 06:26:06,066 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-28) [5ad337f1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 06:26:06,086 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-28) [4aeb6318] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:26:06,089 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-28) [2379f6de] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:26:06,090 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-28) [2379f6de] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 06:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [2e0e0b36] Autorecovering 1 hosts >2013-03-08 06:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [2e0e0b36] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [5df615ce] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [5df615ce] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [5df615ce] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4c58010e >2013-03-08 06:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [5df615ce] FINISH, SetVdsStatusVDSCommand, log id: 4c58010e >2013-03-08 06:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [5df615ce] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3f48c359 >2013-03-08 06:30:00,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [5df615ce] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5fc329dc >2013-03-08 06:30:00,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [5df615ce] FINISH, GetHardwareInfoVDSCommand, log id: 5fc329dc >2013-03-08 06:30:00,097 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-42) [7cd910a7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:30:00,099 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-42) [fa44798] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:30:00,101 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [fa44798] FINISH, ActivateVdsVDSCommand, log id: 3f48c359 >2013-03-08 06:30:00,120 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [fa44798] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:30:00,121 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-42) [fa44798] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 06:30:00,122 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [fa44798] Autorecovering 1 storage domains >2013-03-08 06:30:00,122 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [fa44798] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 06:30:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [21ef9e22] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:30:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [21ef9e22] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 6:30 AM >2013-03-08 06:30:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [21ef9e22] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 6:30 AM >2013-03-08 06:30:00,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [4895537] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7453d009 >2013-03-08 06:30:00,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [4895537] FINISH, GetHardwareInfoVDSCommand, log id: 7453d009 >2013-03-08 06:30:00,338 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-19) [50e593c5] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 06:30:00,342 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-19) [2ba97c6c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 06:30:00,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4d8fdcca >2013-03-08 06:30:33,728 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-99) [409ef71f] Start refreshing all users data >2013-03-08 06:30:34,653 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-49) [c9ee581] Attempting to update VMs/Templates Ovf. >2013-03-08 06:30:34,654 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-49) [c9ee581] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 06:30:34,661 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-49) [c9ee581] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 06:30:34,661 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-49) [c9ee581] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 06:30:34,664 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-49) [c9ee581] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 06:30:34,664 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-49) [c9ee581] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 06:30:34,665 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-49) [c9ee581] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 06:31:06,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 06:31:06,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:31:06,335 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 06:31:06,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 4d8fdcca >2013-03-08 06:31:06,339 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [b460a0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:31:06,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [b460a0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 942833b >2013-03-08 06:31:06,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [b460a0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 06:31:06,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [b460a0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 942833b >2013-03-08 06:31:06,356 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [b460a0] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 06:31:06,356 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [2ba97c6c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 06:31:06,366 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-19) [67cabb93] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:31:06,367 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [67cabb93] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 138f97aa >2013-03-08 06:31:06,368 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [67cabb93] FINISH, SetVdsStatusVDSCommand, log id: 138f97aa >2013-03-08 06:31:06,375 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-19) [67cabb93] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 06:31:06,395 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-19) [54ffc5e0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:31:06,398 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-19) [47d9290b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:31:06,399 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-19) [47d9290b] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 06:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [638bb2a] Autorecovering 1 hosts >2013-03-08 06:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [638bb2a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [516d75e2] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [516d75e2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [516d75e2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5d01caa2 >2013-03-08 06:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [516d75e2] FINISH, SetVdsStatusVDSCommand, log id: 5d01caa2 >2013-03-08 06:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [516d75e2] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 9060e08 >2013-03-08 06:35:00,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [516d75e2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1334c214 >2013-03-08 06:35:00,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [516d75e2] FINISH, GetHardwareInfoVDSCommand, log id: 1334c214 >2013-03-08 06:35:00,195 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-73) [18b1e1d3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:35:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-73) [2ce5aa47] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:35:00,199 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [2ce5aa47] FINISH, ActivateVdsVDSCommand, log id: 9060e08 >2013-03-08 06:35:00,218 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [2ce5aa47] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:35:00,219 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-73) [2ce5aa47] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 06:35:00,220 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [2ce5aa47] Autorecovering 1 storage domains >2013-03-08 06:35:00,220 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [2ce5aa47] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 06:35:00,221 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [5b81993a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:35:00,221 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [5b81993a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 6:35 AM >2013-03-08 06:35:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [5b81993a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 6:35 AM >2013-03-08 06:35:00,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [36b909f5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 32e0e355 >2013-03-08 06:35:00,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [36b909f5] FINISH, GetHardwareInfoVDSCommand, log id: 32e0e355 >2013-03-08 06:35:00,644 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-90) [13106948] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 06:35:00,647 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-90) [7263c50c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 06:35:00,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 25efe788 >2013-03-08 06:36:05,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 06:36:05,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:36:05,645 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 06:36:05,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 25efe788 >2013-03-08 06:36:05,649 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [1abe48cc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:36:05,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1abe48cc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 113fd66b >2013-03-08 06:36:05,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [1abe48cc] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 06:36:05,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1abe48cc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 113fd66b >2013-03-08 06:36:05,668 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [7263c50c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 06:36:05,677 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-90) [25980836] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:36:05,679 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [25980836] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 11ad993a >2013-03-08 06:36:05,680 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [25980836] FINISH, SetVdsStatusVDSCommand, log id: 11ad993a >2013-03-08 06:36:05,686 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [25980836] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 06:36:05,705 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-90) [41c3d600] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:36:05,708 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-90) [51204c0f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:36:05,709 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-90) [51204c0f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 06:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [547a2bbf] Autorecovering 1 hosts >2013-03-08 06:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [547a2bbf] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [7afddd45] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [7afddd45] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [7afddd45] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 108ecf87 >2013-03-08 06:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [7afddd45] FINISH, SetVdsStatusVDSCommand, log id: 108ecf87 >2013-03-08 06:40:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-78) [7afddd45] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7e06ec2e >2013-03-08 06:40:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [7afddd45] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 731135d1 >2013-03-08 06:40:00,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [7afddd45] FINISH, GetHardwareInfoVDSCommand, log id: 731135d1 >2013-03-08 06:40:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-78) [76cc47c5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:40:00,206 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-78) [1cc19ead] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:40:00,208 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-78) [1cc19ead] FINISH, ActivateVdsVDSCommand, log id: 7e06ec2e >2013-03-08 06:40:00,227 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [1cc19ead] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:40:00,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-78) [1cc19ead] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 06:40:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [1cc19ead] Autorecovering 1 storage domains >2013-03-08 06:40:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [1cc19ead] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 06:40:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [f887394] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:40:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [f887394] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 6:40 AM >2013-03-08 06:40:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [f887394] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 6:40 AM >2013-03-08 06:40:02,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [56ef369] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2ea56264 >2013-03-08 06:40:02,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [56ef369] FINISH, GetHardwareInfoVDSCommand, log id: 2ea56264 >2013-03-08 06:40:02,949 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-7) [60bc23d5] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 06:40:02,952 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-7) [7f8d8877] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 06:40:02,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7f17740b >2013-03-08 06:41:06,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 06:41:06,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:41:06,940 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 06:41:06,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 7f17740b >2013-03-08 06:41:06,943 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [6a2890f3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:41:06,957 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [6a2890f3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 75b09c46 >2013-03-08 06:41:06,957 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [6a2890f3] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 06:41:06,957 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [6a2890f3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 75b09c46 >2013-03-08 06:41:06,960 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [6a2890f3] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 06:41:06,963 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [7f8d8877] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 06:41:06,974 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-7) [49f382e4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:41:06,975 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [49f382e4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: a490c54 >2013-03-08 06:41:06,976 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [49f382e4] FINISH, SetVdsStatusVDSCommand, log id: a490c54 >2013-03-08 06:41:06,983 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-7) [49f382e4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 06:41:07,012 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-7) [7142595b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:41:07,014 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-7) [4576a52f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:41:07,016 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) [4576a52f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 06:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [433bdf74] Autorecovering 1 hosts >2013-03-08 06:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [433bdf74] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [2a153970] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [2a153970] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [2a153970] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2e9e2b1e >2013-03-08 06:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [2a153970] FINISH, SetVdsStatusVDSCommand, log id: 2e9e2b1e >2013-03-08 06:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-83) [2a153970] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 62d8ef56 >2013-03-08 06:45:00,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [2a153970] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 646e13d6 >2013-03-08 06:45:00,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [2a153970] FINISH, GetHardwareInfoVDSCommand, log id: 646e13d6 >2013-03-08 06:45:00,262 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-83) [4ad00e85] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:45:00,264 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-83) [9f47303] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:45:00,266 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-83) [9f47303] FINISH, ActivateVdsVDSCommand, log id: 62d8ef56 >2013-03-08 06:45:00,285 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [9f47303] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:45:00,286 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-83) [9f47303] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 06:45:00,287 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [9f47303] Autorecovering 1 storage domains >2013-03-08 06:45:00,287 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [9f47303] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 06:45:00,288 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [45011c9f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:45:00,288 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [45011c9f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 6:45 AM >2013-03-08 06:45:00,291 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [45011c9f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 6:45 AM >2013-03-08 06:45:01,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [785b3a66] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3c6d802f >2013-03-08 06:45:01,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [785b3a66] FINISH, GetHardwareInfoVDSCommand, log id: 3c6d802f >2013-03-08 06:45:01,265 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-22) [750623d0] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 06:45:01,269 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-22) [26b0d9d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 06:45:01,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6caf6aa3 >2013-03-08 06:46:05,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 06:46:05,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:46:05,247 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 06:46:05,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 6caf6aa3 >2013-03-08 06:46:05,250 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [1e34874e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:46:05,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [1e34874e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 50fcfff9 >2013-03-08 06:46:05,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [1e34874e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 06:46:05,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [1e34874e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 50fcfff9 >2013-03-08 06:46:05,267 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [1e34874e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 06:46:05,268 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [26b0d9d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 06:46:05,277 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-22) [2174c337] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:46:05,285 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [2174c337] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5643cfc3 >2013-03-08 06:46:05,286 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [2174c337] FINISH, SetVdsStatusVDSCommand, log id: 5643cfc3 >2013-03-08 06:46:05,292 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [2174c337] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 06:46:05,314 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [70ac3c25] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:46:05,316 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [55006d3f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:46:05,318 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-22) [55006d3f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 06:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [3884e2a8] Autorecovering 1 hosts >2013-03-08 06:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [3884e2a8] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [1a1abaec] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [1a1abaec] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [1a1abaec] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1703de99 >2013-03-08 06:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [1a1abaec] FINISH, SetVdsStatusVDSCommand, log id: 1703de99 >2013-03-08 06:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [1a1abaec] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 52b44b47 >2013-03-08 06:50:00,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [1a1abaec] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 58c27a7d >2013-03-08 06:50:00,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [1a1abaec] FINISH, GetHardwareInfoVDSCommand, log id: 58c27a7d >2013-03-08 06:50:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [4406f12] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:50:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [2f663729] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:50:00,205 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [2f663729] FINISH, ActivateVdsVDSCommand, log id: 52b44b47 >2013-03-08 06:50:00,225 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [2f663729] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:50:00,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-8) [2f663729] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 06:50:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [2f663729] Autorecovering 1 storage domains >2013-03-08 06:50:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [2f663729] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 06:50:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [6579d7aa] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:50:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [6579d7aa] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 6:50 AM >2013-03-08 06:50:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [6579d7aa] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 6:50 AM >2013-03-08 06:50:02,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [7fbdf728] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 170910b7 >2013-03-08 06:50:02,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [7fbdf728] FINISH, GetHardwareInfoVDSCommand, log id: 170910b7 >2013-03-08 06:50:02,560 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-92) [69ab7e66] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 06:50:02,563 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-92) [4df0273f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 06:50:02,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5fc5dc6 >2013-03-08 06:51:08,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 06:51:08,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:51:08,560 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 06:51:08,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 5fc5dc6 >2013-03-08 06:51:08,564 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [7858450c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:51:08,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [7858450c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6f60398d >2013-03-08 06:51:08,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [7858450c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 06:51:08,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [7858450c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6f60398d >2013-03-08 06:51:08,580 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [7858450c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 06:51:08,581 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [4df0273f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 06:51:08,591 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-92) [47f52f70] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:51:08,592 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [47f52f70] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 64b74df9 >2013-03-08 06:51:08,594 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [47f52f70] FINISH, SetVdsStatusVDSCommand, log id: 64b74df9 >2013-03-08 06:51:08,600 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-92) [47f52f70] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 06:51:08,620 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-92) [1942fed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:51:08,623 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-92) [2f044ed4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:51:08,624 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-92) [2f044ed4] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 06:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [2ac2a35] Autorecovering 1 hosts >2013-03-08 06:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [2ac2a35] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [1a56ae42] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [1a56ae42] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [1a56ae42] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2223ee22 >2013-03-08 06:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [1a56ae42] FINISH, SetVdsStatusVDSCommand, log id: 2223ee22 >2013-03-08 06:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [1a56ae42] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 596fe45d >2013-03-08 06:55:00,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [1a56ae42] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1100bd16 >2013-03-08 06:55:00,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [1a56ae42] FINISH, GetHardwareInfoVDSCommand, log id: 1100bd16 >2013-03-08 06:55:00,210 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [4f2573dc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:55:00,213 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [45fe4d3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:55:00,215 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [45fe4d3] FINISH, ActivateVdsVDSCommand, log id: 596fe45d >2013-03-08 06:55:00,238 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [45fe4d3] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 06:55:00,239 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [45fe4d3] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 06:55:00,240 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [45fe4d3] Autorecovering 1 storage domains >2013-03-08 06:55:00,240 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [45fe4d3] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 06:55:00,241 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [7a566d95] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:55:00,241 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [7a566d95] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 6:55 AM >2013-03-08 06:55:00,244 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [7a566d95] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 6:55 AM >2013-03-08 06:55:02,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [4d8c6a6d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7e637ca7 >2013-03-08 06:55:02,832 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [4d8c6a6d] FINISH, GetHardwareInfoVDSCommand, log id: 7e637ca7 >2013-03-08 06:55:02,869 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-10) [63d6252b] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 06:55:02,872 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-10) [227cb7af] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 06:55:02,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 443ab292 >2013-03-08 06:56:08,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 06:56:08,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 06:56:08,907 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 06:56:08,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 443ab292 >2013-03-08 06:56:08,911 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [644df02b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 06:56:08,926 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [644df02b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6ce5ae99 >2013-03-08 06:56:08,926 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [644df02b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 06:56:08,926 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [644df02b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6ce5ae99 >2013-03-08 06:56:08,929 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [644df02b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 06:56:08,929 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [227cb7af] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 06:56:08,939 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-10) [216a1bb4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:56:08,940 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [216a1bb4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7c6d8f9 >2013-03-08 06:56:08,942 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [216a1bb4] FINISH, SetVdsStatusVDSCommand, log id: 7c6d8f9 >2013-03-08 06:56:08,948 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-10) [216a1bb4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 06:56:08,969 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-10) [13dc7eec] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:56:08,971 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-10) [52ed299a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 06:56:08,973 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-10) [52ed299a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 07:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [438879d2] Autorecovering 1 hosts >2013-03-08 07:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [438879d2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [3d15d823] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [3d15d823] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [3d15d823] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7df5fd11 >2013-03-08 07:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [3d15d823] FINISH, SetVdsStatusVDSCommand, log id: 7df5fd11 >2013-03-08 07:00:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-57) [3d15d823] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 24ed8b8b >2013-03-08 07:00:00,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [3d15d823] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7f31fdc7 >2013-03-08 07:00:00,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [3d15d823] FINISH, GetHardwareInfoVDSCommand, log id: 7f31fdc7 >2013-03-08 07:00:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-57) [224799f6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:00:00,207 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-57) [2215fd4b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:00:00,209 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-57) [2215fd4b] FINISH, ActivateVdsVDSCommand, log id: 24ed8b8b >2013-03-08 07:00:00,239 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [2215fd4b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:00:00,240 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-57) [2215fd4b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 07:00:00,241 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [2215fd4b] Autorecovering 1 storage domains >2013-03-08 07:00:00,241 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [2215fd4b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 07:00:00,242 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [7161258c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:00:00,242 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [7161258c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 7:00 AM >2013-03-08 07:00:00,250 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [7161258c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 7:00 AM >2013-03-08 07:00:00,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [2e7f5c74] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4a16a11 >2013-03-08 07:00:00,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [2e7f5c74] FINISH, GetHardwareInfoVDSCommand, log id: 4a16a11 >2013-03-08 07:00:00,309 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-94) [51336fe] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 07:00:00,312 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-94) [9a7240c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 07:00:00,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7b9633f8 >2013-03-08 07:01:06,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 07:01:06,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:01:06,297 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 07:01:06,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 7b9633f8 >2013-03-08 07:01:06,301 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [4d5eefdc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:01:06,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [4d5eefdc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 136f695c >2013-03-08 07:01:06,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [4d5eefdc] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 07:01:06,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [4d5eefdc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 136f695c >2013-03-08 07:01:06,317 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [4d5eefdc] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 07:01:06,317 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [9a7240c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 07:01:06,327 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-94) [5080767d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:01:06,329 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [5080767d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3539294b >2013-03-08 07:01:06,330 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [5080767d] FINISH, SetVdsStatusVDSCommand, log id: 3539294b >2013-03-08 07:01:06,335 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-94) [5080767d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 07:01:06,356 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [1af84b85] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:01:06,359 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [29e0b9b9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:01:06,360 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-94) [29e0b9b9] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 07:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [409ef71f] Autorecovering 1 hosts >2013-03-08 07:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [409ef71f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-99) [bf84bb8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-99) [bf84bb8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [bf84bb8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 66769efd >2013-03-08 07:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [bf84bb8] FINISH, SetVdsStatusVDSCommand, log id: 66769efd >2013-03-08 07:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-99) [bf84bb8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 24edd09e >2013-03-08 07:05:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [bf84bb8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 406203c0 >2013-03-08 07:05:00,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [bf84bb8] FINISH, GetHardwareInfoVDSCommand, log id: 406203c0 >2013-03-08 07:05:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-99) [720c368e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:05:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-99) [573ae0c0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:05:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-99) [573ae0c0] FINISH, ActivateVdsVDSCommand, log id: 24edd09e >2013-03-08 07:05:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-99) [573ae0c0] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:05:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-99) [573ae0c0] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 07:05:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [573ae0c0] Autorecovering 1 storage domains >2013-03-08 07:05:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [573ae0c0] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 07:05:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-99) [651ec7ed] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:05:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-99) [651ec7ed] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 7:05 AM >2013-03-08 07:05:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-99) [651ec7ed] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 7:05 AM >2013-03-08 07:05:00,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [75ba1457] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6a9aefff >2013-03-08 07:05:00,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [75ba1457] FINISH, GetHardwareInfoVDSCommand, log id: 6a9aefff >2013-03-08 07:05:00,596 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-5) [53508891] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 07:05:00,600 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-5) [2b8b7db2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 07:05:00,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6e76cf24 >2013-03-08 07:06:06,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 07:06:06,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:06:06,648 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 07:06:06,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 6e76cf24 >2013-03-08 07:06:06,652 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [2931817f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:06:06,666 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [2931817f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 35bbc8cc >2013-03-08 07:06:06,666 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [2931817f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 07:06:06,666 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [2931817f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 35bbc8cc >2013-03-08 07:06:06,668 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [2931817f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 07:06:06,669 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [2b8b7db2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 07:06:06,678 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-5) [b0af5a3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:06:06,680 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [b0af5a3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3c8d0ad1 >2013-03-08 07:06:06,681 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [b0af5a3] FINISH, SetVdsStatusVDSCommand, log id: 3c8d0ad1 >2013-03-08 07:06:06,687 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-5) [b0af5a3] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 07:06:06,707 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-5) [3062da73] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:06:06,710 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-5) [49119c74] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:06:06,711 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-5) [49119c74] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 07:10:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [316d0cb8] Autorecovering 1 hosts >2013-03-08 07:10:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [316d0cb8] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [91235ad] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [91235ad] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [91235ad] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5aa8df78 >2013-03-08 07:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [91235ad] FINISH, SetVdsStatusVDSCommand, log id: 5aa8df78 >2013-03-08 07:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-41) [91235ad] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 77f680dd >2013-03-08 07:10:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [91235ad] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 14aed06a >2013-03-08 07:10:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [91235ad] FINISH, GetHardwareInfoVDSCommand, log id: 14aed06a >2013-03-08 07:10:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-41) [7293da20] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:10:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-41) [532f7553] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:10:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-41) [532f7553] FINISH, ActivateVdsVDSCommand, log id: 77f680dd >2013-03-08 07:10:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [532f7553] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:10:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-41) [532f7553] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 07:10:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [532f7553] Autorecovering 1 storage domains >2013-03-08 07:10:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [532f7553] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 07:10:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [53b7b56] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:10:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [53b7b56] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 7:10 AM >2013-03-08 07:10:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [53b7b56] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 7:10 AM >2013-03-08 07:10:00,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [2385e91c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 19e0f8c9 >2013-03-08 07:10:00,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [2385e91c] FINISH, GetHardwareInfoVDSCommand, log id: 19e0f8c9 >2013-03-08 07:10:00,959 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-81) [fdc2a18] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 07:10:00,962 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-81) [621b1a33] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 07:10:00,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 60d3c5cf >2013-03-08 07:11:05,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 07:11:05,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:11:05,949 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 07:11:05,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 60d3c5cf >2013-03-08 07:11:05,953 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [3b05c614] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:11:05,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [3b05c614] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3c29db16 >2013-03-08 07:11:05,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [3b05c614] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 07:11:05,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [3b05c614] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3c29db16 >2013-03-08 07:11:05,969 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [3b05c614] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 07:11:05,970 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [621b1a33] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 07:11:05,979 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-81) [29dacc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:11:05,981 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [29dacc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7d27cc41 >2013-03-08 07:11:05,982 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [29dacc] FINISH, SetVdsStatusVDSCommand, log id: 7d27cc41 >2013-03-08 07:11:05,988 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-81) [29dacc] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 07:11:06,009 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-81) [68ed2cec] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:11:06,011 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-81) [2fe8822] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:11:06,013 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-81) [2fe8822] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 07:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-71) [34890b99] Autorecovering 1 hosts >2013-03-08 07:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-71) [34890b99] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-71) [103d8503] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-71) [103d8503] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-71) [103d8503] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 42115805 >2013-03-08 07:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-71) [103d8503] FINISH, SetVdsStatusVDSCommand, log id: 42115805 >2013-03-08 07:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-71) [103d8503] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7ab87cc4 >2013-03-08 07:15:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-71) [103d8503] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 251d317e >2013-03-08 07:15:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-71) [103d8503] FINISH, GetHardwareInfoVDSCommand, log id: 251d317e >2013-03-08 07:15:00,195 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-71) [1e79a283] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:15:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-71) [11b1be5d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:15:00,199 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-71) [11b1be5d] FINISH, ActivateVdsVDSCommand, log id: 7ab87cc4 >2013-03-08 07:15:00,219 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-71) [11b1be5d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:15:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-71) [11b1be5d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 07:15:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-71) [11b1be5d] Autorecovering 1 storage domains >2013-03-08 07:15:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-71) [11b1be5d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 07:15:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-71) [8126383] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:15:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-71) [8126383] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 7:15 AM >2013-03-08 07:15:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-71) [8126383] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 7:15 AM >2013-03-08 07:15:00,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [128fb420] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 596242b1 >2013-03-08 07:15:00,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [128fb420] FINISH, GetHardwareInfoVDSCommand, log id: 596242b1 >2013-03-08 07:15:00,297 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-63) [65255f04] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 07:15:00,300 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-63) [58e94845] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 07:15:00,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 390145ae >2013-03-08 07:16:05,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 07:16:05,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:16:05,277 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 07:16:05,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 390145ae >2013-03-08 07:16:05,281 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [1cd7e64] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:16:05,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1cd7e64] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 519f7743 >2013-03-08 07:16:05,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [1cd7e64] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 07:16:05,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1cd7e64] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 519f7743 >2013-03-08 07:16:05,297 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [1cd7e64] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 07:16:05,297 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [58e94845] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 07:16:05,308 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-63) [1abccda7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:16:05,309 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [1abccda7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5b9fcbb8 >2013-03-08 07:16:05,310 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [1abccda7] FINISH, SetVdsStatusVDSCommand, log id: 5b9fcbb8 >2013-03-08 07:16:05,316 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-63) [1abccda7] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 07:16:05,336 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-63) [75565599] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:16:05,339 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-63) [6c01ec2e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:16:05,340 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-63) [6c01ec2e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 07:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [2353e7ee] Autorecovering 1 hosts >2013-03-08 07:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [2353e7ee] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [4be9d1cb] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [4be9d1cb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [4be9d1cb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4ff378c9 >2013-03-08 07:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [4be9d1cb] FINISH, SetVdsStatusVDSCommand, log id: 4ff378c9 >2013-03-08 07:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-33) [4be9d1cb] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 79c84e6e >2013-03-08 07:20:00,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [4be9d1cb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4ccb8a5 >2013-03-08 07:20:00,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [4be9d1cb] FINISH, GetHardwareInfoVDSCommand, log id: 4ccb8a5 >2013-03-08 07:20:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-33) [4da16bb2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:20:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-33) [329110ca] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:20:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-33) [329110ca] FINISH, ActivateVdsVDSCommand, log id: 79c84e6e >2013-03-08 07:20:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [329110ca] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:20:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-33) [329110ca] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 07:20:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [329110ca] Autorecovering 1 storage domains >2013-03-08 07:20:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [329110ca] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 07:20:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [2c2b5c5f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:20:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [2c2b5c5f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 7:20 AM >2013-03-08 07:20:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [2c2b5c5f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 7:20 AM >2013-03-08 07:20:02,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [1c68a073] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7e620605 >2013-03-08 07:20:02,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [1c68a073] FINISH, GetHardwareInfoVDSCommand, log id: 7e620605 >2013-03-08 07:20:02,588 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-38) [f6c83ae] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 07:20:02,591 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-38) [7d25e3fb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 07:20:02,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3170ed60 >2013-03-08 07:21:06,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 07:21:06,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:21:06,636 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 07:21:06,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 3170ed60 >2013-03-08 07:21:06,639 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [7364c409] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:21:06,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [7364c409] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 650f5912 >2013-03-08 07:21:06,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [7364c409] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 07:21:06,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [7364c409] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 650f5912 >2013-03-08 07:21:06,661 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [7364c409] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 07:21:06,661 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [7d25e3fb] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 07:21:06,671 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-38) [1256a0d7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:21:06,672 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [1256a0d7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5947b855 >2013-03-08 07:21:06,674 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [1256a0d7] FINISH, SetVdsStatusVDSCommand, log id: 5947b855 >2013-03-08 07:21:06,680 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-38) [1256a0d7] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 07:21:06,700 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-38) [63a4eecc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:21:06,703 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-38) [643024c1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:21:06,704 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-38) [643024c1] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 07:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [6fbec909] Autorecovering 1 hosts >2013-03-08 07:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [6fbec909] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [29fd570e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [29fd570e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [29fd570e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 46e9d87f >2013-03-08 07:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [29fd570e] FINISH, SetVdsStatusVDSCommand, log id: 46e9d87f >2013-03-08 07:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-21) [29fd570e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 28a16cb0 >2013-03-08 07:25:00,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [29fd570e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3b6131d0 >2013-03-08 07:25:00,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [29fd570e] FINISH, GetHardwareInfoVDSCommand, log id: 3b6131d0 >2013-03-08 07:25:00,268 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-21) [11cfba32] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:25:00,271 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-21) [4b16abb5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:25:00,272 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-21) [4b16abb5] FINISH, ActivateVdsVDSCommand, log id: 28a16cb0 >2013-03-08 07:25:00,291 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [4b16abb5] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:25:00,292 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-21) [4b16abb5] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 07:25:00,293 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [4b16abb5] Autorecovering 1 storage domains >2013-03-08 07:25:00,293 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [4b16abb5] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 07:25:00,294 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [54a87db] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:25:00,294 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [54a87db] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 7:25 AM >2013-03-08 07:25:00,297 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [54a87db] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 7:25 AM >2013-03-08 07:25:00,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [1b9ae193] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 44f06b26 >2013-03-08 07:25:00,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [1b9ae193] FINISH, GetHardwareInfoVDSCommand, log id: 44f06b26 >2013-03-08 07:25:00,952 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-18) [1eeece84] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 07:25:00,985 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-18) [19f3a6f2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 07:25:00,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 24bbf7fe >2013-03-08 07:26:07,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 07:26:07,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:26:07,028 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 07:26:07,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 24bbf7fe >2013-03-08 07:26:07,031 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [1805c959] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:26:07,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [1805c959] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 72a9679c >2013-03-08 07:26:07,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [1805c959] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 07:26:07,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [1805c959] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 72a9679c >2013-03-08 07:26:07,047 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [1805c959] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 07:26:07,047 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [19f3a6f2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 07:26:07,057 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-18) [3ae6c154] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:26:07,059 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [3ae6c154] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7860d533 >2013-03-08 07:26:07,060 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [3ae6c154] FINISH, SetVdsStatusVDSCommand, log id: 7860d533 >2013-03-08 07:26:07,066 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [3ae6c154] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 07:26:07,086 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [67b31fbc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:26:07,088 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [12f9356a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:26:07,089 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) [12f9356a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 07:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [a99f831] Autorecovering 1 hosts >2013-03-08 07:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [a99f831] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [683b2d7c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [683b2d7c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [683b2d7c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2ffc9185 >2013-03-08 07:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [683b2d7c] FINISH, SetVdsStatusVDSCommand, log id: 2ffc9185 >2013-03-08 07:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [683b2d7c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 23c1249 >2013-03-08 07:30:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [683b2d7c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1fc05be8 >2013-03-08 07:30:00,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [683b2d7c] FINISH, GetHardwareInfoVDSCommand, log id: 1fc05be8 >2013-03-08 07:30:00,259 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-96) [357ecee8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:30:00,261 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-96) [18bd5ed7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:30:00,263 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [18bd5ed7] FINISH, ActivateVdsVDSCommand, log id: 23c1249 >2013-03-08 07:30:00,282 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [18bd5ed7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:30:00,283 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-96) [18bd5ed7] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 07:30:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [18bd5ed7] Autorecovering 1 storage domains >2013-03-08 07:30:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [18bd5ed7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 07:30:00,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [1d77d63b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:30:00,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [1d77d63b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 7:30 AM >2013-03-08 07:30:00,288 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [1d77d63b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 7:30 AM >2013-03-08 07:30:01,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [62cc6576] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 412c8223 >2013-03-08 07:30:01,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [62cc6576] FINISH, GetHardwareInfoVDSCommand, log id: 412c8223 >2013-03-08 07:30:01,335 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-36) [3e738ee9] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 07:30:01,338 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-36) [17e7af99] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 07:30:01,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 33fbba95 >2013-03-08 07:30:33,730 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-67) [6ffac8dc] Start refreshing all users data >2013-03-08 07:30:34,665 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [621c65a2] Attempting to update VMs/Templates Ovf. >2013-03-08 07:30:34,666 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [621c65a2] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 07:30:34,667 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [621c65a2] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 07:30:34,667 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [621c65a2] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 07:30:34,668 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [621c65a2] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 07:30:34,668 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [621c65a2] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 07:30:34,668 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [621c65a2] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 07:31:05,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 07:31:05,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:31:05,365 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 07:31:05,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 33fbba95 >2013-03-08 07:31:05,369 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [5889c161] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:31:05,383 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5889c161] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6f1172cf >2013-03-08 07:31:05,383 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [5889c161] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 07:31:05,383 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5889c161] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6f1172cf >2013-03-08 07:31:05,385 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [5889c161] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 07:31:05,385 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [17e7af99] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 07:31:05,395 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-36) [6519846a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:31:05,396 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [6519846a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 21f987d7 >2013-03-08 07:31:05,398 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [6519846a] FINISH, SetVdsStatusVDSCommand, log id: 21f987d7 >2013-03-08 07:31:05,404 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [6519846a] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 07:31:05,432 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [785f768a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:31:05,435 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [2f367b32] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:31:05,436 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-36) [2f367b32] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 07:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) Autorecovering 1 hosts >2013-03-08 07:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [319d3cc1] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [319d3cc1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [319d3cc1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4423e6c7 >2013-03-08 07:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [319d3cc1] FINISH, SetVdsStatusVDSCommand, log id: 4423e6c7 >2013-03-08 07:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-1) [319d3cc1] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 22fccbea >2013-03-08 07:35:00,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [319d3cc1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3321211a >2013-03-08 07:35:00,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [319d3cc1] FINISH, GetHardwareInfoVDSCommand, log id: 3321211a >2013-03-08 07:35:00,185 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-1) [2cf61e99] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:35:00,188 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-1) [7c2b450a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:35:00,189 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-1) [7c2b450a] FINISH, ActivateVdsVDSCommand, log id: 22fccbea >2013-03-08 07:35:00,209 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [7c2b450a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:35:00,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-1) [7c2b450a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 07:35:00,211 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [7c2b450a] Autorecovering 1 storage domains >2013-03-08 07:35:00,211 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [7c2b450a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 07:35:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [6c97e9a6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:35:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [6c97e9a6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 7:35 AM >2013-03-08 07:35:00,215 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [6c97e9a6] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 7:35 AM >2013-03-08 07:35:02,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [2379f6de] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2e98ea6c >2013-03-08 07:35:02,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [2379f6de] FINISH, GetHardwareInfoVDSCommand, log id: 2e98ea6c >2013-03-08 07:35:02,685 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-28) [74628d85] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 07:35:02,688 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-28) [374cb03e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 07:35:02,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2f3d4d6b >2013-03-08 07:36:06,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 07:36:06,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:36:06,668 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 07:36:06,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2f3d4d6b >2013-03-08 07:36:06,672 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [7815c905] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:36:06,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [7815c905] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6c1ce7bf >2013-03-08 07:36:06,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [7815c905] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 07:36:06,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [7815c905] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6c1ce7bf >2013-03-08 07:36:06,690 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-48) [374cb03e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 07:36:06,701 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-28) [42bd7e28] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:36:06,702 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [42bd7e28] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6acd07cd >2013-03-08 07:36:06,703 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [42bd7e28] FINISH, SetVdsStatusVDSCommand, log id: 6acd07cd >2013-03-08 07:36:06,710 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-28) [42bd7e28] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 07:36:06,729 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-28) [2ca52c74] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:36:06,732 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-28) [6ae64941] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:36:06,733 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-28) [6ae64941] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 07:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [1c2fa1f] Autorecovering 1 hosts >2013-03-08 07:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [1c2fa1f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [621c2be1] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [621c2be1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [621c2be1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 405c34d2 >2013-03-08 07:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [621c2be1] FINISH, SetVdsStatusVDSCommand, log id: 405c34d2 >2013-03-08 07:40:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [621c2be1] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: d2d375c >2013-03-08 07:40:00,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [621c2be1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 70b9c31f >2013-03-08 07:40:00,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [621c2be1] FINISH, GetHardwareInfoVDSCommand, log id: 70b9c31f >2013-03-08 07:40:00,939 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-69) [6b744da] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:40:00,941 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-69) [70bfa0ea] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:40:00,942 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [70bfa0ea] FINISH, ActivateVdsVDSCommand, log id: d2d375c >2013-03-08 07:40:00,962 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [70bfa0ea] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:40:00,963 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-69) [70bfa0ea] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 07:40:00,964 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [70bfa0ea] Autorecovering 1 storage domains >2013-03-08 07:40:00,964 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [70bfa0ea] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 07:40:00,965 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [356167d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:40:00,965 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [356167d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 7:40 AM >2013-03-08 07:40:00,968 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [356167d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 7:40 AM >2013-03-08 07:40:00,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-40) [327c1650] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 646b50ff >2013-03-08 07:40:01,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-40) [327c1650] FINISH, GetHardwareInfoVDSCommand, log id: 646b50ff >2013-03-08 07:40:01,040 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-40) [3613d1df] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 07:40:01,043 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-40) [107d33b2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 07:40:01,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 662c9330 >2013-03-08 07:41:06,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 07:41:06,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:41:06,049 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 07:41:06,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 662c9330 >2013-03-08 07:41:06,053 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [1e6c6bec] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:41:06,067 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1e6c6bec] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5ce6fba5 >2013-03-08 07:41:06,067 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [1e6c6bec] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 07:41:06,067 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [1e6c6bec] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5ce6fba5 >2013-03-08 07:41:06,069 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [1e6c6bec] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 07:41:06,069 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [107d33b2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 07:41:06,079 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-40) [47771789] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:41:06,080 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-40) [47771789] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 310bd52a >2013-03-08 07:41:06,082 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-40) [47771789] FINISH, SetVdsStatusVDSCommand, log id: 310bd52a >2013-03-08 07:41:06,088 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-40) [47771789] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 07:41:06,108 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-40) [259fe025] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:41:06,111 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-40) [90e1ee3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:41:06,112 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-40) [90e1ee3] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 07:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [465df3e5] Autorecovering 1 hosts >2013-03-08 07:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [465df3e5] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [2df0a6b4] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [2df0a6b4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [2df0a6b4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4b47e8cf >2013-03-08 07:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [2df0a6b4] FINISH, SetVdsStatusVDSCommand, log id: 4b47e8cf >2013-03-08 07:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-35) [2df0a6b4] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 16995ff3 >2013-03-08 07:45:00,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [2df0a6b4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5c0a6cf2 >2013-03-08 07:45:00,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [2df0a6b4] FINISH, GetHardwareInfoVDSCommand, log id: 5c0a6cf2 >2013-03-08 07:45:00,086 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-35) [9cfbbc7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:45:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-35) [52824c6a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:45:00,090 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-35) [52824c6a] FINISH, ActivateVdsVDSCommand, log id: 16995ff3 >2013-03-08 07:45:00,109 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [52824c6a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:45:00,110 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-35) [52824c6a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 07:45:00,112 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [52824c6a] Autorecovering 1 storage domains >2013-03-08 07:45:00,112 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [52824c6a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 07:45:00,113 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [693e8288] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:45:00,113 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [693e8288] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 7:45 AM >2013-03-08 07:45:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [693e8288] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 7:45 AM >2013-03-08 07:45:00,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [37d30159] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6a7ca2b8 >2013-03-08 07:45:00,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [37d30159] FINISH, GetHardwareInfoVDSCommand, log id: 6a7ca2b8 >2013-03-08 07:45:00,354 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-59) [4603bd46] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 07:45:00,357 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-59) [708d3cc1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 07:45:00,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 40307e9c >2013-03-08 07:46:05,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 07:46:05,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:46:05,656 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 07:46:05,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 40307e9c >2013-03-08 07:46:05,659 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [75ef8ebb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:46:05,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [75ef8ebb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 21ecf2a6 >2013-03-08 07:46:05,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [75ef8ebb] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 07:46:05,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [75ef8ebb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 21ecf2a6 >2013-03-08 07:46:05,676 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [75ef8ebb] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 07:46:05,676 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [708d3cc1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 07:46:05,685 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-59) [23bf6130] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:46:05,687 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [23bf6130] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 42db1fc5 >2013-03-08 07:46:05,688 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [23bf6130] FINISH, SetVdsStatusVDSCommand, log id: 42db1fc5 >2013-03-08 07:46:05,693 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [23bf6130] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 07:46:05,713 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [15e977a8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:46:05,715 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [11989d9a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:46:05,717 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-59) [11989d9a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 07:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [4a818f99] Autorecovering 1 hosts >2013-03-08 07:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [4a818f99] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-25) [18b7222e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-25) [18b7222e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [18b7222e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 20e90dd0 >2013-03-08 07:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [18b7222e] FINISH, SetVdsStatusVDSCommand, log id: 20e90dd0 >2013-03-08 07:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-25) [18b7222e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 43994561 >2013-03-08 07:50:00,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [18b7222e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4129b025 >2013-03-08 07:50:00,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [18b7222e] FINISH, GetHardwareInfoVDSCommand, log id: 4129b025 >2013-03-08 07:50:00,257 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-25) [7a59257e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:50:00,259 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-25) [567c492e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:50:00,260 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-25) [567c492e] FINISH, ActivateVdsVDSCommand, log id: 43994561 >2013-03-08 07:50:00,280 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-25) [567c492e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:50:00,280 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-25) [567c492e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 07:50:00,282 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [567c492e] Autorecovering 1 storage domains >2013-03-08 07:50:00,282 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [567c492e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 07:50:00,282 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-25) [77295e42] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:50:00,283 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-25) [77295e42] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 7:50 AM >2013-03-08 07:50:00,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-25) [77295e42] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 7:50 AM >2013-03-08 07:50:02,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [1fa2973c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 14725ca5 >2013-03-08 07:50:02,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [1fa2973c] FINISH, GetHardwareInfoVDSCommand, log id: 14725ca5 >2013-03-08 07:50:02,962 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-91) [5958572d] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 07:50:02,965 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-91) [60def8e4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 07:50:02,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 72f8e0fb >2013-03-08 07:51:06,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 07:51:06,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-48) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:51:06,971 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-48) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 07:51:06,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 72f8e0fb >2013-03-08 07:51:06,975 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-48) [4b92899c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:51:06,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [4b92899c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c225c53 >2013-03-08 07:51:06,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-48) [4b92899c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 07:51:06,990 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-48) [4b92899c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c225c53 >2013-03-08 07:51:06,992 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-48) [4b92899c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 07:51:06,992 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [60def8e4] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 07:51:07,008 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-91) [4f287c41] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:51:07,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [4f287c41] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 13ecc9c2 >2013-03-08 07:51:07,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [4f287c41] FINISH, SetVdsStatusVDSCommand, log id: 13ecc9c2 >2013-03-08 07:51:07,017 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-91) [4f287c41] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 07:51:07,038 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-91) [3371f104] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:51:07,041 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-91) [66d5d2f7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:51:07,043 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-91) [66d5d2f7] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 07:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [101fdc3f] Autorecovering 1 hosts >2013-03-08 07:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [101fdc3f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [414804ee] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [414804ee] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [414804ee] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 490b36d9 >2013-03-08 07:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [414804ee] FINISH, SetVdsStatusVDSCommand, log id: 490b36d9 >2013-03-08 07:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-75) [414804ee] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 85acf66 >2013-03-08 07:55:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [414804ee] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 730ac824 >2013-03-08 07:55:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [414804ee] FINISH, GetHardwareInfoVDSCommand, log id: 730ac824 >2013-03-08 07:55:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [5966eda] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:55:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [341147d3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:55:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-75) [341147d3] FINISH, ActivateVdsVDSCommand, log id: 85acf66 >2013-03-08 07:55:00,222 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [341147d3] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 07:55:00,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-75) [341147d3] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 07:55:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [341147d3] Autorecovering 1 storage domains >2013-03-08 07:55:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [341147d3] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 07:55:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [787a75db] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:55:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [787a75db] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 7:55 AM >2013-03-08 07:55:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [787a75db] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 7:55 AM >2013-03-08 07:55:01,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [52ed299a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4e99d716 >2013-03-08 07:55:01,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [52ed299a] FINISH, GetHardwareInfoVDSCommand, log id: 4e99d716 >2013-03-08 07:55:01,287 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-10) [794d5a8d] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 07:55:01,290 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-10) [2ccd4cd5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 07:55:01,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 77bf43fe >2013-03-08 07:56:06,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 07:56:06,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 07:56:06,342 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 07:56:06,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 77bf43fe >2013-03-08 07:56:06,345 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [2d45b3fe] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 07:56:06,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [2d45b3fe] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 655af821 >2013-03-08 07:56:06,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [2d45b3fe] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 07:56:06,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [2d45b3fe] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 655af821 >2013-03-08 07:56:06,362 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [2d45b3fe] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 07:56:06,363 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [2ccd4cd5] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 07:56:06,372 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-10) [32a7057f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:56:06,374 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [32a7057f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 28a6e0c3 >2013-03-08 07:56:06,375 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [32a7057f] FINISH, SetVdsStatusVDSCommand, log id: 28a6e0c3 >2013-03-08 07:56:06,381 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-10) [32a7057f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 07:56:06,401 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-10) [794d0c51] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:56:06,404 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-10) [2219fb2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 07:56:06,405 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-10) [2219fb2] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 08:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [29e0b9b9] Autorecovering 1 hosts >2013-03-08 08:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [29e0b9b9] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [25998f3e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [25998f3e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [25998f3e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 38dea6e4 >2013-03-08 08:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [25998f3e] FINISH, SetVdsStatusVDSCommand, log id: 38dea6e4 >2013-03-08 08:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [25998f3e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: a4efdbe >2013-03-08 08:00:00,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [25998f3e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 12226b9 >2013-03-08 08:00:00,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [25998f3e] FINISH, GetHardwareInfoVDSCommand, log id: 12226b9 >2013-03-08 08:00:00,186 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [63118571] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:00:00,189 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [607e4a08] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:00:00,190 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [607e4a08] FINISH, ActivateVdsVDSCommand, log id: a4efdbe >2013-03-08 08:00:00,209 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [607e4a08] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:00:00,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-94) [607e4a08] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 08:00:00,211 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [607e4a08] Autorecovering 1 storage domains >2013-03-08 08:00:00,211 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [607e4a08] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 08:00:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [47fee925] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:00:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [47fee925] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 8:00 AM >2013-03-08 08:00:00,215 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [47fee925] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 8:00 AM >2013-03-08 08:00:00,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [9f7f384] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: ed66e9 >2013-03-08 08:00:00,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [9f7f384] FINISH, GetHardwareInfoVDSCommand, log id: ed66e9 >2013-03-08 08:00:00,652 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-48) [6eb1326f] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 08:00:00,655 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-48) [635e963f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 08:00:00,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 233b629b >2013-03-08 08:01:05,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 08:01:05,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:01:05,708 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 08:01:05,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 233b629b >2013-03-08 08:01:05,711 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [7659569a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:01:05,725 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [7659569a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1e61a25e >2013-03-08 08:01:05,725 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [7659569a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 08:01:05,725 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [7659569a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1e61a25e >2013-03-08 08:01:05,727 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [7659569a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 08:01:05,728 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [635e963f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 08:01:05,743 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-48) [16540a8c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:01:05,745 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [16540a8c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4e69a9ff >2013-03-08 08:01:05,746 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [16540a8c] FINISH, SetVdsStatusVDSCommand, log id: 4e69a9ff >2013-03-08 08:01:05,751 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-48) [16540a8c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 08:01:05,771 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-48) [47874e2a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:01:05,774 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-48) [23d73427] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:01:05,775 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-48) [23d73427] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 08:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [35d63329] Autorecovering 1 hosts >2013-03-08 08:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [35d63329] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-95) [74980b3c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-95) [74980b3c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [74980b3c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 39172a9 >2013-03-08 08:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [74980b3c] FINISH, SetVdsStatusVDSCommand, log id: 39172a9 >2013-03-08 08:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-95) [74980b3c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4b31f65f >2013-03-08 08:05:00,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [74980b3c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 256faa9f >2013-03-08 08:05:00,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [74980b3c] FINISH, GetHardwareInfoVDSCommand, log id: 256faa9f >2013-03-08 08:05:00,240 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-95) [4f4251f6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:05:00,243 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-95) [68fa5e92] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:05:00,244 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-95) [68fa5e92] FINISH, ActivateVdsVDSCommand, log id: 4b31f65f >2013-03-08 08:05:00,264 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-95) [68fa5e92] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:05:00,264 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-95) [68fa5e92] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 08:05:00,266 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [68fa5e92] Autorecovering 1 storage domains >2013-03-08 08:05:00,266 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [68fa5e92] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 08:05:00,266 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-95) [12230346] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:05:00,267 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-95) [12230346] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 8:05 AM >2013-03-08 08:05:00,269 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-95) [12230346] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 8:05 AM >2013-03-08 08:05:02,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [21ef9e22] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 782b0aab >2013-03-08 08:05:02,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [21ef9e22] FINISH, GetHardwareInfoVDSCommand, log id: 782b0aab >2013-03-08 08:05:03,020 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-42) [55898588] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 08:05:03,024 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-42) [2173dc74] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 08:05:03,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2234fd40 >2013-03-08 08:06:06,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 08:06:06,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:06:06,004 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 08:06:06,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 2234fd40 >2013-03-08 08:06:06,008 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [69c96daf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:06:06,022 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [69c96daf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2aea4994 >2013-03-08 08:06:06,022 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [69c96daf] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 08:06:06,022 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [69c96daf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2aea4994 >2013-03-08 08:06:06,025 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [69c96daf] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 08:06:06,025 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [2173dc74] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 08:06:06,035 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-42) [7b08a451] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:06:06,036 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [7b08a451] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 103f18fb >2013-03-08 08:06:06,037 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [7b08a451] FINISH, SetVdsStatusVDSCommand, log id: 103f18fb >2013-03-08 08:06:06,044 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-42) [7b08a451] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 08:06:06,064 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-42) [1c997bdc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:06:06,066 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-42) [3a7fcf03] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:06:06,068 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-42) [3a7fcf03] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 08:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [47f2e6ff] Autorecovering 1 hosts >2013-03-08 08:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [47f2e6ff] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [22f78445] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [22f78445] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [22f78445] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6399ab38 >2013-03-08 08:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [22f78445] FINISH, SetVdsStatusVDSCommand, log id: 6399ab38 >2013-03-08 08:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [22f78445] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1c01a347 >2013-03-08 08:10:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [22f78445] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4f1cae15 >2013-03-08 08:10:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [22f78445] FINISH, GetHardwareInfoVDSCommand, log id: 4f1cae15 >2013-03-08 08:10:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [455cd57b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:10:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [77d96e7d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:10:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [77d96e7d] FINISH, ActivateVdsVDSCommand, log id: 1c01a347 >2013-03-08 08:10:00,231 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [77d96e7d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:10:00,232 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-52) [77d96e7d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 08:10:00,233 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [77d96e7d] Autorecovering 1 storage domains >2013-03-08 08:10:00,234 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [77d96e7d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 08:10:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [597dc8bb] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:10:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [597dc8bb] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 8:10 AM >2013-03-08 08:10:00,237 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [597dc8bb] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 8:10 AM >2013-03-08 08:10:00,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [7a566d95] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 168cd9f2 >2013-03-08 08:10:00,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [7a566d95] FINISH, GetHardwareInfoVDSCommand, log id: 168cd9f2 >2013-03-08 08:10:00,315 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-43) [72d6c739] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 08:10:00,318 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-43) [531c6552] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 08:10:00,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4406c5fe >2013-03-08 08:11:05,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 08:11:05,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:11:05,310 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 08:11:05,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 4406c5fe >2013-03-08 08:11:05,314 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [58180d98] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:11:05,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [58180d98] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2bc987e2 >2013-03-08 08:11:05,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [58180d98] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 08:11:05,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [58180d98] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2bc987e2 >2013-03-08 08:11:05,330 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [58180d98] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 08:11:05,330 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [531c6552] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 08:11:05,340 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-43) [7e41ee7a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:11:05,342 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [7e41ee7a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3e09b0a6 >2013-03-08 08:11:05,343 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [7e41ee7a] FINISH, SetVdsStatusVDSCommand, log id: 3e09b0a6 >2013-03-08 08:11:05,349 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [7e41ee7a] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 08:11:05,370 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [418a9da] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:11:05,372 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [3126ba3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:11:05,373 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-43) [3126ba3] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 08:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [7692caee] Autorecovering 1 hosts >2013-03-08 08:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [7692caee] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [5b4ec938] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [5b4ec938] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [5b4ec938] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 15e6b073 >2013-03-08 08:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [5b4ec938] FINISH, SetVdsStatusVDSCommand, log id: 15e6b073 >2013-03-08 08:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-65) [5b4ec938] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1394822d >2013-03-08 08:15:00,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [5b4ec938] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 69e4c2dd >2013-03-08 08:15:00,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [5b4ec938] FINISH, GetHardwareInfoVDSCommand, log id: 69e4c2dd >2013-03-08 08:15:00,657 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-65) [30b49c85] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:15:00,659 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-65) [f059c1c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:15:00,661 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-65) [f059c1c] FINISH, ActivateVdsVDSCommand, log id: 1394822d >2013-03-08 08:15:00,680 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [f059c1c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:15:00,681 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-65) [f059c1c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 08:15:00,682 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [f059c1c] Autorecovering 1 storage domains >2013-03-08 08:15:00,682 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [f059c1c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 08:15:00,683 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [45e710d4] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:15:00,683 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [45e710d4] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 8:15 AM >2013-03-08 08:15:00,685 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [45e710d4] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 8:15 AM >2013-03-08 08:15:02,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [6579d7aa] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: cbfde5c >2013-03-08 08:15:02,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [6579d7aa] FINISH, GetHardwareInfoVDSCommand, log id: cbfde5c >2013-03-08 08:15:02,623 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-8) [335cf64f] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 08:15:02,626 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-8) [5554ec5d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 08:15:02,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 664f07fe >2013-03-08 08:16:06,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 08:16:06,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:16:06,642 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 08:16:06,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 664f07fe >2013-03-08 08:16:06,646 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [6031783e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:16:06,660 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6031783e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: cda57dd >2013-03-08 08:16:06,660 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [6031783e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 08:16:06,660 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6031783e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: cda57dd >2013-03-08 08:16:06,662 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [6031783e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 08:16:06,662 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [5554ec5d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 08:16:06,672 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-8) [3336be86] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:16:06,674 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [3336be86] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 105cbcfc >2013-03-08 08:16:06,675 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [3336be86] FINISH, SetVdsStatusVDSCommand, log id: 105cbcfc >2013-03-08 08:16:06,682 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-8) [3336be86] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 08:16:06,708 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [5e3fc512] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:16:06,711 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [e5d01fb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:16:06,712 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-8) [e5d01fb] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 08:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [74e312c1] Autorecovering 1 hosts >2013-03-08 08:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [74e312c1] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [48a6f334] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [48a6f334] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [48a6f334] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7072a010 >2013-03-08 08:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [48a6f334] FINISH, SetVdsStatusVDSCommand, log id: 7072a010 >2013-03-08 08:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-9) [48a6f334] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 688e1c5c >2013-03-08 08:20:00,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [48a6f334] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: f619224 >2013-03-08 08:20:00,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [48a6f334] FINISH, GetHardwareInfoVDSCommand, log id: f619224 >2013-03-08 08:20:00,094 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-9) [585aaa8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:20:00,097 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-9) [63313e61] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:20:00,098 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-9) [63313e61] FINISH, ActivateVdsVDSCommand, log id: 688e1c5c >2013-03-08 08:20:00,117 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [63313e61] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:20:00,118 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-9) [63313e61] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 08:20:00,119 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [63313e61] Autorecovering 1 storage domains >2013-03-08 08:20:00,120 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [63313e61] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 08:20:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [47a66593] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:20:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [47a66593] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 8:20 AM >2013-03-08 08:20:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [47a66593] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 8:20 AM >2013-03-08 08:20:00,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [8598a6c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2bfdb626 >2013-03-08 08:20:00,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [8598a6c] FINISH, GetHardwareInfoVDSCommand, log id: 2bfdb626 >2013-03-08 08:20:00,958 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-2) [57ef5b6c] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 08:20:00,961 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-2) [27555dd9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 08:20:00,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 17626473 >2013-03-08 08:21:04,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 08:21:04,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:21:04,957 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 08:21:04,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 17626473 >2013-03-08 08:21:04,960 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [314e4e12] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:21:04,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [314e4e12] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 722e217c >2013-03-08 08:21:04,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [314e4e12] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 08:21:04,975 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [314e4e12] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 722e217c >2013-03-08 08:21:04,977 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [314e4e12] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 08:21:04,977 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [27555dd9] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 08:21:04,987 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-2) [14c4221c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:21:04,988 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [14c4221c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 33b4620e >2013-03-08 08:21:04,990 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [14c4221c] FINISH, SetVdsStatusVDSCommand, log id: 33b4620e >2013-03-08 08:21:04,996 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [14c4221c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 08:21:05,018 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [4241bbee] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:21:05,020 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [6a49e344] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:21:05,021 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-2) [6a49e344] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 08:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [15343e77] Autorecovering 1 hosts >2013-03-08 08:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [15343e77] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [4ed6bc5] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [4ed6bc5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [4ed6bc5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 75d3a8f8 >2013-03-08 08:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [4ed6bc5] FINISH, SetVdsStatusVDSCommand, log id: 75d3a8f8 >2013-03-08 08:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [4ed6bc5] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1c02d719 >2013-03-08 08:25:00,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [4ed6bc5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5736386e >2013-03-08 08:25:00,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [4ed6bc5] FINISH, GetHardwareInfoVDSCommand, log id: 5736386e >2013-03-08 08:25:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [122a7eec] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:25:00,095 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [1ba945cb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:25:00,097 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [1ba945cb] FINISH, ActivateVdsVDSCommand, log id: 1c02d719 >2013-03-08 08:25:00,116 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [1ba945cb] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:25:00,117 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [1ba945cb] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 08:25:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [1ba945cb] Autorecovering 1 storage domains >2013-03-08 08:25:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [1ba945cb] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 08:25:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [5d0ef834] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:25:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [5d0ef834] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 8:25 AM >2013-03-08 08:25:00,122 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [5d0ef834] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 8:25 AM >2013-03-08 08:25:02,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [43b479b3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 65acf4da >2013-03-08 08:25:02,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [43b479b3] FINISH, GetHardwareInfoVDSCommand, log id: 65acf4da >2013-03-08 08:25:02,265 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-61) [55bbd0ce] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 08:25:02,269 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-61) [20eb418d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 08:25:02,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4f1645f3 >2013-03-08 08:26:07,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 08:26:07,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:26:07,313 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 08:26:07,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 4f1645f3 >2013-03-08 08:26:07,316 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [1f978224] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:26:07,331 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [1f978224] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4c773091 >2013-03-08 08:26:07,331 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [1f978224] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 08:26:07,331 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [1f978224] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4c773091 >2013-03-08 08:26:07,333 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [1f978224] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 08:26:07,333 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [20eb418d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 08:26:07,343 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-61) [501b9a8c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:26:07,345 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [501b9a8c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 602261c8 >2013-03-08 08:26:07,346 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [501b9a8c] FINISH, SetVdsStatusVDSCommand, log id: 602261c8 >2013-03-08 08:26:07,352 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-61) [501b9a8c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 08:26:07,374 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-61) [5190011c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:26:07,377 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-61) [71cd7733] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:26:07,378 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-61) [71cd7733] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 08:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [2f367b32] Autorecovering 1 hosts >2013-03-08 08:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [2f367b32] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [4417aa10] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [4417aa10] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [4417aa10] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2cdd1949 >2013-03-08 08:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [4417aa10] FINISH, SetVdsStatusVDSCommand, log id: 2cdd1949 >2013-03-08 08:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [4417aa10] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 63f95af5 >2013-03-08 08:30:00,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [4417aa10] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 78970e83 >2013-03-08 08:30:00,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [4417aa10] FINISH, GetHardwareInfoVDSCommand, log id: 78970e83 >2013-03-08 08:30:00,532 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [79a63c1f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:30:00,534 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [764861d6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:30:00,536 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [764861d6] FINISH, ActivateVdsVDSCommand, log id: 63f95af5 >2013-03-08 08:30:00,555 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [764861d6] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:30:00,556 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [764861d6] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 08:30:00,557 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [764861d6] Autorecovering 1 storage domains >2013-03-08 08:30:00,557 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [764861d6] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 08:30:00,558 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [5461d040] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:30:00,558 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [5461d040] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 8:30 AM >2013-03-08 08:30:00,561 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [5461d040] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 8:30 AM >2013-03-08 08:30:01,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [29ec8a6a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2a54f63 >2013-03-08 08:30:01,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [29ec8a6a] FINISH, GetHardwareInfoVDSCommand, log id: 2a54f63 >2013-03-08 08:30:01,629 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-46) [6f5b454b] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 08:30:01,632 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-46) [5e829580] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 08:30:01,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5d1b7314 >2013-03-08 08:30:33,732 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-92) [2f044ed4] Start refreshing all users data >2013-03-08 08:30:34,668 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-96) [1d77d63b] Attempting to update VMs/Templates Ovf. >2013-03-08 08:30:34,669 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-96) [1d77d63b] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 08:30:34,670 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-96) [1d77d63b] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 08:30:34,670 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-96) [1d77d63b] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 08:30:34,671 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-96) [1d77d63b] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 08:30:34,671 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-96) [1d77d63b] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 08:30:34,671 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-96) [1d77d63b] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 08:31:05,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 08:31:05,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:31:05,637 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 08:31:05,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 5d1b7314 >2013-03-08 08:31:05,640 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [76384563] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:31:05,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [76384563] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6cb31960 >2013-03-08 08:31:05,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [76384563] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 08:31:05,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [76384563] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6cb31960 >2013-03-08 08:31:05,657 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [76384563] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 08:31:05,657 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [5e829580] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 08:31:05,668 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-46) [4b93aeb0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:31:05,669 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [4b93aeb0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6067caf2 >2013-03-08 08:31:05,670 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [4b93aeb0] FINISH, SetVdsStatusVDSCommand, log id: 6067caf2 >2013-03-08 08:31:05,677 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-46) [4b93aeb0] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 08:31:05,697 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-46) [4a472a03] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:31:05,700 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-46) [7882a1df] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:31:05,701 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-46) [7882a1df] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 08:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [6b04b768] Autorecovering 1 hosts >2013-03-08 08:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [6b04b768] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-76) [41f30b61] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-76) [41f30b61] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [41f30b61] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7421ada7 >2013-03-08 08:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [41f30b61] FINISH, SetVdsStatusVDSCommand, log id: 7421ada7 >2013-03-08 08:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-76) [41f30b61] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 72e4db4d >2013-03-08 08:35:00,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [41f30b61] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 73fb696b >2013-03-08 08:35:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [41f30b61] FINISH, GetHardwareInfoVDSCommand, log id: 73fb696b >2013-03-08 08:35:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-76) [5e4a48b7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:35:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-76) [6db68981] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:35:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-76) [6db68981] FINISH, ActivateVdsVDSCommand, log id: 72e4db4d >2013-03-08 08:35:00,111 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-76) [6db68981] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:35:00,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-76) [6db68981] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 08:35:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [6db68981] Autorecovering 1 storage domains >2013-03-08 08:35:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [6db68981] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 08:35:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-76) [1be500d9] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:35:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-76) [1be500d9] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 8:35 AM >2013-03-08 08:35:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-76) [1be500d9] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 8:35 AM >2013-03-08 08:35:02,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [3858662b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7c7876c9 >2013-03-08 08:35:02,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [3858662b] FINISH, GetHardwareInfoVDSCommand, log id: 7c7876c9 >2013-03-08 08:35:02,974 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-14) [488fd22a] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 08:35:02,978 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-14) [6cd646c1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 08:35:02,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 254be54d >2013-03-08 08:36:06,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 08:36:06,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:36:06,957 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 08:36:06,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 254be54d >2013-03-08 08:36:06,960 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [40ff1d88] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:36:06,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [40ff1d88] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 11274ee7 >2013-03-08 08:36:06,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [40ff1d88] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 08:36:06,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [40ff1d88] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 11274ee7 >2013-03-08 08:36:06,981 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [6cd646c1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 08:36:06,990 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-14) [53ad4929] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:36:06,991 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [53ad4929] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 38c917df >2013-03-08 08:36:06,993 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [53ad4929] FINISH, SetVdsStatusVDSCommand, log id: 38c917df >2013-03-08 08:36:06,998 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-14) [53ad4929] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 08:36:07,018 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-14) [5d052c01] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:36:07,020 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-14) [19f642c4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:36:07,022 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-14) [19f642c4] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 08:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [5bbb7ef8] Autorecovering 1 hosts >2013-03-08 08:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [5bbb7ef8] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [7e8a1a26] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [7e8a1a26] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [7e8a1a26] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2c297607 >2013-03-08 08:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [7e8a1a26] FINISH, SetVdsStatusVDSCommand, log id: 2c297607 >2013-03-08 08:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-24) [7e8a1a26] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 17dabe32 >2013-03-08 08:40:00,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [7e8a1a26] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 36f04c05 >2013-03-08 08:40:00,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [7e8a1a26] FINISH, GetHardwareInfoVDSCommand, log id: 36f04c05 >2013-03-08 08:40:00,307 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-24) [462ba2ea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:40:00,309 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-24) [2ea7cc5d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:40:00,311 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-24) [2ea7cc5d] FINISH, ActivateVdsVDSCommand, log id: 17dabe32 >2013-03-08 08:40:00,330 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [2ea7cc5d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:40:00,331 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-24) [2ea7cc5d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 08:40:00,332 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [2ea7cc5d] Autorecovering 1 storage domains >2013-03-08 08:40:00,332 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [2ea7cc5d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 08:40:00,333 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [633810d2] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:40:00,333 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [633810d2] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 8:40 AM >2013-03-08 08:40:00,336 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [633810d2] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 8:40 AM >2013-03-08 08:40:01,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [231273d2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6c8a6526 >2013-03-08 08:40:01,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [231273d2] FINISH, GetHardwareInfoVDSCommand, log id: 6c8a6526 >2013-03-08 08:40:01,269 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-51) [73fd2c96] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 08:40:01,273 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-51) [98a9f61] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 08:40:01,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5b9b4c39 >2013-03-08 08:41:06,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 08:41:06,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:41:06,257 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 08:41:06,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 5b9b4c39 >2013-03-08 08:41:06,261 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [3844bd7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:41:06,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [3844bd7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 503a0e84 >2013-03-08 08:41:06,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [3844bd7] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 08:41:06,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [3844bd7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 503a0e84 >2013-03-08 08:41:06,278 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [3844bd7] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 08:41:06,279 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [98a9f61] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 08:41:06,289 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-51) [f6b65d6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:41:06,290 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [f6b65d6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3d0e558e >2013-03-08 08:41:06,292 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [f6b65d6] FINISH, SetVdsStatusVDSCommand, log id: 3d0e558e >2013-03-08 08:41:06,297 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-51) [f6b65d6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 08:41:06,318 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-51) [1a26c2a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:41:06,321 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-51) [75c8a919] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:41:06,322 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-51) [75c8a919] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 08:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [46a42c3f] Autorecovering 1 hosts >2013-03-08 08:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [46a42c3f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-53) [5deed15a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-53) [5deed15a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [5deed15a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4ec6987c >2013-03-08 08:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [5deed15a] FINISH, SetVdsStatusVDSCommand, log id: 4ec6987c >2013-03-08 08:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-53) [5deed15a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3cdd2e4a >2013-03-08 08:45:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [5deed15a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 74533bbe >2013-03-08 08:45:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [5deed15a] FINISH, GetHardwareInfoVDSCommand, log id: 74533bbe >2013-03-08 08:45:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-53) [2258edc8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:45:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-53) [75198830] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:45:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-53) [75198830] FINISH, ActivateVdsVDSCommand, log id: 3cdd2e4a >2013-03-08 08:45:00,219 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-53) [75198830] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:45:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-53) [75198830] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 08:45:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [75198830] Autorecovering 1 storage domains >2013-03-08 08:45:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [75198830] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 08:45:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-53) [470c8030] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:45:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-53) [470c8030] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 8:45 AM >2013-03-08 08:45:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-53) [470c8030] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 8:45 AM >2013-03-08 08:45:00,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [53b7b56] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 18d8b2d3 >2013-03-08 08:45:00,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [53b7b56] FINISH, GetHardwareInfoVDSCommand, log id: 18d8b2d3 >2013-03-08 08:45:00,568 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-41) [587e545e] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 08:45:00,572 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-41) [766c3ea8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 08:45:00,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 188bdcea >2013-03-08 08:46:05,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 08:46:05,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:46:05,877 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 08:46:05,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 188bdcea >2013-03-08 08:46:05,880 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [56aacc8b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:46:05,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [56aacc8b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 19009921 >2013-03-08 08:46:05,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [56aacc8b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 08:46:05,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [56aacc8b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 19009921 >2013-03-08 08:46:05,897 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [56aacc8b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 08:46:05,898 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [766c3ea8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 08:46:05,908 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-41) [5fbbf50f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:46:05,909 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [5fbbf50f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 206648fb >2013-03-08 08:46:05,910 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [5fbbf50f] FINISH, SetVdsStatusVDSCommand, log id: 206648fb >2013-03-08 08:46:05,917 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-41) [5fbbf50f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 08:46:05,938 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-41) [42e71749] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:46:05,940 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-41) [17fd5d6f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:46:05,942 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-41) [17fd5d6f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 08:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [533b0fff] Autorecovering 1 hosts >2013-03-08 08:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [533b0fff] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [64668a6] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [64668a6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [64668a6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4461d167 >2013-03-08 08:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [64668a6] FINISH, SetVdsStatusVDSCommand, log id: 4461d167 >2013-03-08 08:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [64668a6] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 43d20a5e >2013-03-08 08:50:00,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [64668a6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4e7440ce >2013-03-08 08:50:00,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [64668a6] FINISH, GetHardwareInfoVDSCommand, log id: 4e7440ce >2013-03-08 08:50:00,227 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [46f3e781] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:50:00,229 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [5f445839] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:50:00,231 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [5f445839] FINISH, ActivateVdsVDSCommand, log id: 43d20a5e >2013-03-08 08:50:00,250 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [5f445839] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:50:00,251 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [5f445839] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 08:50:00,252 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [5f445839] Autorecovering 1 storage domains >2013-03-08 08:50:00,252 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [5f445839] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 08:50:00,253 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [6f15c46e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:50:00,253 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [6f15c46e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 8:50 AM >2013-03-08 08:50:00,256 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [6f15c46e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 8:50 AM >2013-03-08 08:50:00,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [66d5d2f7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 47d6b78e >2013-03-08 08:50:00,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [66d5d2f7] FINISH, GetHardwareInfoVDSCommand, log id: 47d6b78e >2013-03-08 08:50:00,328 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-91) [269c9624] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 08:50:00,331 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-91) [6d98b5b8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 08:50:00,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 11f5afdd >2013-03-08 08:51:06,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 08:51:06,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:51:06,335 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 08:51:06,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 11f5afdd >2013-03-08 08:51:06,339 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [64df0175] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:51:06,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [64df0175] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 524089a1 >2013-03-08 08:51:06,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [64df0175] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 08:51:06,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [64df0175] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 524089a1 >2013-03-08 08:51:06,356 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [64df0175] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 08:51:06,356 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [6d98b5b8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 08:51:06,366 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-91) [31f66993] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:51:06,367 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [31f66993] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4d06ca83 >2013-03-08 08:51:06,369 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [31f66993] FINISH, SetVdsStatusVDSCommand, log id: 4d06ca83 >2013-03-08 08:51:06,375 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-91) [31f66993] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 08:51:06,395 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-91) [429189a9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:51:06,397 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-91) [558864c3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:51:06,399 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-91) [558864c3] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 08:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [68cfb3b9] Autorecovering 1 hosts >2013-03-08 08:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [68cfb3b9] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [5022895f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [5022895f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [5022895f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 159b0523 >2013-03-08 08:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [5022895f] FINISH, SetVdsStatusVDSCommand, log id: 159b0523 >2013-03-08 08:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [5022895f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1d91ba00 >2013-03-08 08:55:00,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [5022895f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 153a0880 >2013-03-08 08:55:00,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [5022895f] FINISH, GetHardwareInfoVDSCommand, log id: 153a0880 >2013-03-08 08:55:00,193 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [3c710cf7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:55:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [33f2bd73] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:55:00,197 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [33f2bd73] FINISH, ActivateVdsVDSCommand, log id: 1d91ba00 >2013-03-08 08:55:00,217 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [33f2bd73] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 08:55:00,217 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [33f2bd73] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 08:55:00,219 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [33f2bd73] Autorecovering 1 storage domains >2013-03-08 08:55:00,219 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [33f2bd73] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 08:55:00,219 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [2cb65d8f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:55:00,220 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [2cb65d8f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 8:55 AM >2013-03-08 08:55:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [2cb65d8f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 8:55 AM >2013-03-08 08:55:00,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [2c2b5c5f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4021d2ea >2013-03-08 08:55:00,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [2c2b5c5f] FINISH, GetHardwareInfoVDSCommand, log id: 4021d2ea >2013-03-08 08:55:00,639 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-33) [56d441c9] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 08:55:00,643 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-33) [1dfa2e32] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 08:55:00,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7a79070 >2013-03-08 08:56:06,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 08:56:06,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 08:56:06,721 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 08:56:06,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 7a79070 >2013-03-08 08:56:06,725 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [23289b98] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 08:56:06,739 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [23289b98] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 70690c5f >2013-03-08 08:56:06,739 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [23289b98] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 08:56:06,739 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [23289b98] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 70690c5f >2013-03-08 08:56:06,741 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [23289b98] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 08:56:06,742 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [1dfa2e32] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 08:56:06,751 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-33) [3f690c9c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:56:06,753 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [3f690c9c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6e5c5ae3 >2013-03-08 08:56:06,754 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [3f690c9c] FINISH, SetVdsStatusVDSCommand, log id: 6e5c5ae3 >2013-03-08 08:56:06,760 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-33) [3f690c9c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 08:56:06,782 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-33) [6e37501f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:56:06,784 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-33) [73c289b8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 08:56:06,785 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-33) [73c289b8] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 09:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [2616e83f] Autorecovering 1 hosts >2013-03-08 09:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [2616e83f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [25edd6cf] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [25edd6cf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [25edd6cf] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2873508c >2013-03-08 09:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [25edd6cf] FINISH, SetVdsStatusVDSCommand, log id: 2873508c >2013-03-08 09:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [25edd6cf] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 37d2a124 >2013-03-08 09:00:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [25edd6cf] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1042977 >2013-03-08 09:00:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [25edd6cf] FINISH, GetHardwareInfoVDSCommand, log id: 1042977 >2013-03-08 09:00:00,258 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-16) [6674252b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:00:00,261 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-16) [18e699a4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:00:00,262 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [18e699a4] FINISH, ActivateVdsVDSCommand, log id: 37d2a124 >2013-03-08 09:00:00,281 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [18e699a4] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:00:00,282 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-16) [18e699a4] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 09:00:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [18e699a4] Autorecovering 1 storage domains >2013-03-08 09:00:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [18e699a4] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 09:00:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [5c8aad20] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:00:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [5c8aad20] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 9:00 AM >2013-03-08 09:00:00,287 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [5c8aad20] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 9:00 AM >2013-03-08 09:00:00,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [5f6cb773] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 75c79852 >2013-03-08 09:00:00,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [5f6cb773] FINISH, GetHardwareInfoVDSCommand, log id: 75c79852 >2013-03-08 09:00:01,031 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-82) [76a8d144] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 09:00:01,035 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-82) [71e288a9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 09:00:01,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5ae2eaf7 >2013-03-08 09:01:06,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 09:01:06,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:01:06,055 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 09:01:06,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 5ae2eaf7 >2013-03-08 09:01:06,059 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [5dc8207c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:01:06,073 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5dc8207c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1bf4de8e >2013-03-08 09:01:06,073 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [5dc8207c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 09:01:06,073 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5dc8207c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1bf4de8e >2013-03-08 09:01:06,075 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [5dc8207c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 09:01:06,076 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [71e288a9] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 09:01:06,085 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-82) [4d3f3a1f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:01:06,087 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [4d3f3a1f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 44bd0c9d >2013-03-08 09:01:06,088 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [4d3f3a1f] FINISH, SetVdsStatusVDSCommand, log id: 44bd0c9d >2013-03-08 09:01:06,094 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-82) [4d3f3a1f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 09:01:06,114 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-82) [147e2551] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:01:06,117 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-82) [68b8a2e0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:01:06,118 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-82) [68b8a2e0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 09:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [a7eaa44] Autorecovering 1 hosts >2013-03-08 09:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [a7eaa44] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [7e146cba] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [7e146cba] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [7e146cba] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: c158a03 >2013-03-08 09:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [7e146cba] FINISH, SetVdsStatusVDSCommand, log id: c158a03 >2013-03-08 09:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-64) [7e146cba] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4e6d91c9 >2013-03-08 09:05:00,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [7e146cba] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3fc82b6b >2013-03-08 09:05:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [7e146cba] FINISH, GetHardwareInfoVDSCommand, log id: 3fc82b6b >2013-03-08 09:05:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-64) [2fde9fbf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:05:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-64) [1d5e9dcf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:05:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-64) [1d5e9dcf] FINISH, ActivateVdsVDSCommand, log id: 4e6d91c9 >2013-03-08 09:05:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [1d5e9dcf] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:05:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-64) [1d5e9dcf] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 09:05:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [1d5e9dcf] Autorecovering 1 storage domains >2013-03-08 09:05:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [1d5e9dcf] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 09:05:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [6f33e1c0] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:05:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [6f33e1c0] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 9:05 AM >2013-03-08 09:05:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [6f33e1c0] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 9:05 AM >2013-03-08 09:05:00,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [4576a52f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: a30ef8f >2013-03-08 09:05:00,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [4576a52f] FINISH, GetHardwareInfoVDSCommand, log id: a30ef8f >2013-03-08 09:05:00,360 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-7) [49989001] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 09:05:00,364 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-7) [410c49db] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 09:05:00,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7fbc3677 >2013-03-08 09:06:03,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 09:06:03,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:06:03,346 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 09:06:03,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 7fbc3677 >2013-03-08 09:06:03,350 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [32129f43] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:06:03,364 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [32129f43] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 17c1e158 >2013-03-08 09:06:03,365 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [32129f43] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 09:06:03,365 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [32129f43] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 17c1e158 >2013-03-08 09:06:03,367 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [32129f43] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 09:06:03,367 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [410c49db] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 09:06:03,377 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-7) [463cfa21] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:06:03,378 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [463cfa21] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5416848d >2013-03-08 09:06:03,379 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [463cfa21] FINISH, SetVdsStatusVDSCommand, log id: 5416848d >2013-03-08 09:06:03,386 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-7) [463cfa21] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 09:06:03,406 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-7) [54a08b18] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:06:03,408 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-7) [4d747221] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:06:03,409 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) [4d747221] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 09:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [3126ba3] Autorecovering 1 hosts >2013-03-08 09:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [3126ba3] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [644a4207] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [644a4207] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [644a4207] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7206fb57 >2013-03-08 09:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [644a4207] FINISH, SetVdsStatusVDSCommand, log id: 7206fb57 >2013-03-08 09:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [644a4207] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6600de1d >2013-03-08 09:10:00,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [644a4207] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 56b339c9 >2013-03-08 09:10:00,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [644a4207] FINISH, GetHardwareInfoVDSCommand, log id: 56b339c9 >2013-03-08 09:10:00,189 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [713ce40d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:10:00,191 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [3e467666] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:10:00,193 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [3e467666] FINISH, ActivateVdsVDSCommand, log id: 6600de1d >2013-03-08 09:10:00,212 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [3e467666] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:10:00,213 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [3e467666] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 09:10:00,214 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [3e467666] Autorecovering 1 storage domains >2013-03-08 09:10:00,214 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [3e467666] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 09:10:00,215 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [6097ad7] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:10:00,215 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [6097ad7] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 9:10 AM >2013-03-08 09:10:00,218 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [6097ad7] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 9:10 AM >2013-03-08 09:10:00,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [693e8288] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3cf99722 >2013-03-08 09:10:00,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [693e8288] FINISH, GetHardwareInfoVDSCommand, log id: 3cf99722 >2013-03-08 09:10:00,658 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-35) [79ce736] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 09:10:00,662 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-35) [7027caaf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 09:10:00,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3624d3c6 >2013-03-08 09:11:04,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 09:11:04,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:11:04,677 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 09:11:04,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 3624d3c6 >2013-03-08 09:11:04,711 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [315a94f4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:11:04,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [315a94f4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6a624aa3 >2013-03-08 09:11:04,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [315a94f4] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 09:11:04,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [315a94f4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6a624aa3 >2013-03-08 09:11:04,730 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [315a94f4] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 09:11:04,730 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [7027caaf] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 09:11:04,740 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-35) [452cbaa3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:11:04,741 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [452cbaa3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4920b444 >2013-03-08 09:11:04,742 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [452cbaa3] FINISH, SetVdsStatusVDSCommand, log id: 4920b444 >2013-03-08 09:11:04,749 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-35) [452cbaa3] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 09:11:04,769 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-35) [7b2c9949] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:11:04,772 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-35) [5a01051a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:11:04,773 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-35) [5a01051a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 09:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [99e81db] Autorecovering 1 hosts >2013-03-08 09:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [99e81db] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [7dbc10c6] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [7dbc10c6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [7dbc10c6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7d46d5c4 >2013-03-08 09:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [7dbc10c6] FINISH, SetVdsStatusVDSCommand, log id: 7d46d5c4 >2013-03-08 09:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-77) [7dbc10c6] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 32e252cd >2013-03-08 09:15:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [7dbc10c6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3caa0e39 >2013-03-08 09:15:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [7dbc10c6] FINISH, GetHardwareInfoVDSCommand, log id: 3caa0e39 >2013-03-08 09:15:00,257 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-77) [17b55f8d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:15:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-77) [24cbdd7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:15:00,261 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-77) [24cbdd7] FINISH, ActivateVdsVDSCommand, log id: 32e252cd >2013-03-08 09:15:00,281 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [24cbdd7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:15:00,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-77) [24cbdd7] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 09:15:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [24cbdd7] Autorecovering 1 storage domains >2013-03-08 09:15:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [24cbdd7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 09:15:00,283 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [6892803f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:15:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [6892803f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 9:15 AM >2013-03-08 09:15:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [6892803f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 9:15 AM >2013-03-08 09:15:01,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [3a7fcf03] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 439b6466 >2013-03-08 09:15:01,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [3a7fcf03] FINISH, GetHardwareInfoVDSCommand, log id: 439b6466 >2013-03-08 09:15:02,022 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-42) [2afe3050] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 09:15:02,026 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-42) [564c2c9e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 09:15:02,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 150147f >2013-03-08 09:16:09,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 09:16:09,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:16:09,009 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 09:16:09,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 150147f >2013-03-08 09:16:09,013 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [2795ea28] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:16:09,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [2795ea28] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4732fbbd >2013-03-08 09:16:09,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [2795ea28] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 09:16:09,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [2795ea28] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4732fbbd >2013-03-08 09:16:09,029 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [2795ea28] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 09:16:09,030 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [564c2c9e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 09:16:09,039 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-42) [66494296] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:16:09,041 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [66494296] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6a69b0de >2013-03-08 09:16:09,042 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [66494296] FINISH, SetVdsStatusVDSCommand, log id: 6a69b0de >2013-03-08 09:16:09,050 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-42) [66494296] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 09:16:09,071 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-42) [e7cf293] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:16:09,074 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-42) [1d287d1c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:16:09,075 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-42) [1d287d1c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 09:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [f887394] Autorecovering 1 hosts >2013-03-08 09:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [f887394] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [4828bcca] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [4828bcca] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [4828bcca] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 518a19c4 >2013-03-08 09:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [4828bcca] FINISH, SetVdsStatusVDSCommand, log id: 518a19c4 >2013-03-08 09:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-78) [4828bcca] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 61fb453c >2013-03-08 09:20:00,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [4828bcca] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1bfb2e70 >2013-03-08 09:20:00,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [4828bcca] FINISH, GetHardwareInfoVDSCommand, log id: 1bfb2e70 >2013-03-08 09:20:00,243 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-78) [5bd7e177] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:20:00,246 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-78) [3c41586e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:20:00,248 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-78) [3c41586e] FINISH, ActivateVdsVDSCommand, log id: 61fb453c >2013-03-08 09:20:00,268 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [3c41586e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:20:00,271 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [3c41586e] Autorecovering 1 storage domains >2013-03-08 09:20:00,271 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [3c41586e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 09:20:00,272 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [2cbaa5b5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:20:00,272 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [2cbaa5b5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 9:20 AM >2013-03-08 09:20:00,275 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [2cbaa5b5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 9:20 AM >2013-03-08 09:20:00,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [36e1c345] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 74fe3582 >2013-03-08 09:20:00,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [36e1c345] FINISH, GetHardwareInfoVDSCommand, log id: 74fe3582 >2013-03-08 09:20:00,346 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-34) [3b94abec] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 09:20:00,349 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-34) [534fbf19] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 09:20:00,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 345d786a >2013-03-08 09:21:06,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 09:21:06,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:21:06,362 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 09:21:06,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 345d786a >2013-03-08 09:21:06,365 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [11aee0af] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:21:06,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [11aee0af] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 77c82f3c >2013-03-08 09:21:06,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [11aee0af] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 09:21:06,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [11aee0af] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 77c82f3c >2013-03-08 09:21:06,381 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [11aee0af] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 09:21:06,381 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [534fbf19] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 09:21:06,391 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-34) [129d0b4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:21:06,393 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [129d0b4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 673dbe2d >2013-03-08 09:21:06,394 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [129d0b4] FINISH, SetVdsStatusVDSCommand, log id: 673dbe2d >2013-03-08 09:21:06,400 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-34) [129d0b4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 09:21:06,421 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-34) [6b18db1c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:21:06,423 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-34) [d1e7f84] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:21:06,424 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) [d1e7f84] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 09:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [47fee925] Autorecovering 1 hosts >2013-03-08 09:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [47fee925] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [75fc6552] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [75fc6552] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [75fc6552] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 526e46d0 >2013-03-08 09:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [75fc6552] FINISH, SetVdsStatusVDSCommand, log id: 526e46d0 >2013-03-08 09:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [75fc6552] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5e0634f7 >2013-03-08 09:25:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [75fc6552] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7ab9c449 >2013-03-08 09:25:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [75fc6552] FINISH, GetHardwareInfoVDSCommand, log id: 7ab9c449 >2013-03-08 09:25:00,259 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [7d8ec21c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:25:00,262 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [539e03b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:25:00,263 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [539e03b] FINISH, ActivateVdsVDSCommand, log id: 5e0634f7 >2013-03-08 09:25:00,282 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [539e03b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:25:00,283 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-94) [539e03b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 09:25:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [539e03b] Autorecovering 1 storage domains >2013-03-08 09:25:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [539e03b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 09:25:00,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [45f5e7be] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:25:00,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [45f5e7be] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 9:25 AM >2013-03-08 09:25:00,288 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [45f5e7be] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 9:25 AM >2013-03-08 09:25:00,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-85) [6853da68] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 23018cb8 >2013-03-08 09:25:00,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-85) [6853da68] FINISH, GetHardwareInfoVDSCommand, log id: 23018cb8 >2013-03-08 09:25:00,668 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-85) [265fce28] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 09:25:00,671 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-85) [4c275661] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 09:25:00,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1a0f0c19 >2013-03-08 09:26:04,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 09:26:04,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:26:04,706 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 09:26:04,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 1a0f0c19 >2013-03-08 09:26:04,709 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [72b93274] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:26:04,723 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [72b93274] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 69707aa9 >2013-03-08 09:26:04,723 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [72b93274] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 09:26:04,723 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [72b93274] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 69707aa9 >2013-03-08 09:26:04,726 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [72b93274] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 09:26:04,726 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [4c275661] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 09:26:04,735 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-85) [14bb3fc1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:26:04,737 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-85) [14bb3fc1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2891ac83 >2013-03-08 09:26:04,738 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-85) [14bb3fc1] FINISH, SetVdsStatusVDSCommand, log id: 2891ac83 >2013-03-08 09:26:04,744 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-85) [14bb3fc1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 09:26:04,765 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-85) [2279f65f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:26:04,768 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-85) [7d17da86] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:26:04,769 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-85) [7d17da86] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 09:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [47f066f5] Autorecovering 1 hosts >2013-03-08 09:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [47f066f5] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-39) [719ba745] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-39) [719ba745] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [719ba745] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 42529ca3 >2013-03-08 09:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [719ba745] FINISH, SetVdsStatusVDSCommand, log id: 42529ca3 >2013-03-08 09:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-39) [719ba745] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5f390925 >2013-03-08 09:30:00,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [719ba745] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5404e17 >2013-03-08 09:30:00,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [719ba745] FINISH, GetHardwareInfoVDSCommand, log id: 5404e17 >2013-03-08 09:30:00,206 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-39) [2ee74a3c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:30:00,209 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-39) [71fcbe2d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:30:00,210 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-39) [71fcbe2d] FINISH, ActivateVdsVDSCommand, log id: 5f390925 >2013-03-08 09:30:00,230 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-39) [71fcbe2d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:30:00,231 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-39) [71fcbe2d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 09:30:00,232 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [71fcbe2d] Autorecovering 1 storage domains >2013-03-08 09:30:00,232 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [71fcbe2d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 09:30:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-39) [33958dd7] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:30:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-39) [33958dd7] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 9:30 AM >2013-03-08 09:30:00,236 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-39) [33958dd7] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 9:30 AM >2013-03-08 09:30:01,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [49119c74] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6f2418c3 >2013-03-08 09:30:01,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [49119c74] FINISH, GetHardwareInfoVDSCommand, log id: 6f2418c3 >2013-03-08 09:30:02,015 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-5) [1cbb5afa] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 09:30:02,018 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-5) [c1d61df] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 09:30:02,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4fa1c074 >2013-03-08 09:30:33,735 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-20) [3a756cd6] Start refreshing all users data >2013-03-08 09:30:34,672 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [633810d2] Attempting to update VMs/Templates Ovf. >2013-03-08 09:30:34,673 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [633810d2] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 09:30:34,674 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [633810d2] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 09:30:34,674 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [633810d2] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 09:30:34,675 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [633810d2] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 09:30:34,675 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [633810d2] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 09:30:34,675 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [633810d2] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 09:31:07,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 09:31:07,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:31:07,070 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 09:31:07,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 4fa1c074 >2013-03-08 09:31:07,074 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [4393ceac] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:31:07,088 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [4393ceac] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6c26dcff >2013-03-08 09:31:07,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [4393ceac] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 09:31:07,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [4393ceac] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6c26dcff >2013-03-08 09:31:07,091 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [4393ceac] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 09:31:07,092 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [c1d61df] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 09:31:07,102 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-5) [46bed059] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:31:07,103 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [46bed059] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3a9ebbf1 >2013-03-08 09:31:07,104 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [46bed059] FINISH, SetVdsStatusVDSCommand, log id: 3a9ebbf1 >2013-03-08 09:31:07,111 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-5) [46bed059] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 09:31:07,131 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-5) [2fffc925] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:31:07,134 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-5) [1f3593ef] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:31:07,135 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-5) [1f3593ef] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 09:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [45011c9f] Autorecovering 1 hosts >2013-03-08 09:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [45011c9f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [1d01efa9] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [1d01efa9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [1d01efa9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 48fb3c01 >2013-03-08 09:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [1d01efa9] FINISH, SetVdsStatusVDSCommand, log id: 48fb3c01 >2013-03-08 09:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-83) [1d01efa9] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2e075e92 >2013-03-08 09:35:00,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [1d01efa9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 73f3fb54 >2013-03-08 09:35:00,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [1d01efa9] FINISH, GetHardwareInfoVDSCommand, log id: 73f3fb54 >2013-03-08 09:35:00,310 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-83) [e241989] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:35:00,313 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-83) [2b21549] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:35:00,314 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-83) [2b21549] FINISH, ActivateVdsVDSCommand, log id: 2e075e92 >2013-03-08 09:35:00,333 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [2b21549] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:35:00,334 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-83) [2b21549] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 09:35:00,335 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [2b21549] Autorecovering 1 storage domains >2013-03-08 09:35:00,335 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [2b21549] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 09:35:00,336 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [5c67c943] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:35:00,336 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [5c67c943] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 9:35 AM >2013-03-08 09:35:00,339 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [5c67c943] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 9:35 AM >2013-03-08 09:35:01,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [3127cf5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 48214d9c >2013-03-08 09:35:01,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [3127cf5] FINISH, GetHardwareInfoVDSCommand, log id: 48214d9c >2013-03-08 09:35:01,375 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-93) [15903d28] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 09:35:01,378 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-93) [72f909a7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 09:35:01,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5381ed92 >2013-03-08 09:36:05,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 09:36:05,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:36:05,376 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 09:36:05,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 5381ed92 >2013-03-08 09:36:05,380 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [4f77d52f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:36:05,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [4f77d52f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2001e67a >2013-03-08 09:36:05,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [4f77d52f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 09:36:05,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [4f77d52f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2001e67a >2013-03-08 09:36:05,403 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [4f77d52f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 09:36:05,403 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [72f909a7] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 09:36:05,413 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-93) [57b17157] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:36:05,414 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [57b17157] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 35e91dcf >2013-03-08 09:36:05,415 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [57b17157] FINISH, SetVdsStatusVDSCommand, log id: 35e91dcf >2013-03-08 09:36:05,421 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-93) [57b17157] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 09:36:05,463 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [5d7d5b6e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:36:05,465 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [f047e82] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:36:05,467 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-93) [f047e82] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 09:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [45e710d4] Autorecovering 1 hosts >2013-03-08 09:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [45e710d4] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [a78a41e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [a78a41e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [a78a41e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 434ce46c >2013-03-08 09:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [a78a41e] FINISH, SetVdsStatusVDSCommand, log id: 434ce46c >2013-03-08 09:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-65) [a78a41e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 188b3bb >2013-03-08 09:40:00,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [a78a41e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7fca0add >2013-03-08 09:40:00,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [a78a41e] FINISH, GetHardwareInfoVDSCommand, log id: 7fca0add >2013-03-08 09:40:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-65) [258a0fbc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:40:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-65) [741a34b3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:40:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-65) [741a34b3] FINISH, ActivateVdsVDSCommand, log id: 188b3bb >2013-03-08 09:40:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [741a34b3] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:40:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-65) [741a34b3] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 09:40:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [741a34b3] Autorecovering 1 storage domains >2013-03-08 09:40:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [741a34b3] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 09:40:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [617179c6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:40:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [617179c6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 9:40 AM >2013-03-08 09:40:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [617179c6] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 9:40 AM >2013-03-08 09:40:02,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [33958dd7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 804d789 >2013-03-08 09:40:02,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [33958dd7] FINISH, GetHardwareInfoVDSCommand, log id: 804d789 >2013-03-08 09:40:02,741 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-39) [d7f8496] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 09:40:02,744 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-39) [797eece2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 09:40:02,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3cb3d8ce >2013-03-08 09:41:06,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 09:41:06,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:41:06,741 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 09:41:06,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 3cb3d8ce >2013-03-08 09:41:06,745 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [33efaf54] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:41:06,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [33efaf54] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 48392821 >2013-03-08 09:41:06,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [33efaf54] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 09:41:06,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [33efaf54] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 48392821 >2013-03-08 09:41:06,771 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [797eece2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 09:41:06,780 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-39) [4e546f1e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:41:06,782 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [4e546f1e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7add3b1 >2013-03-08 09:41:06,783 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [4e546f1e] FINISH, SetVdsStatusVDSCommand, log id: 7add3b1 >2013-03-08 09:41:06,790 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-39) [4e546f1e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 09:41:06,810 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-39) [4f82c38a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:41:06,812 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-39) [2e6cd64] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:41:06,814 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-39) [2e6cd64] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 09:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) [78dbf18] Autorecovering 1 hosts >2013-03-08 09:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) [78dbf18] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-11) [9c00753] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-11) [9c00753] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [9c00753] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 499ffc6f >2013-03-08 09:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [9c00753] FINISH, SetVdsStatusVDSCommand, log id: 499ffc6f >2013-03-08 09:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-11) [9c00753] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 237da3f1 >2013-03-08 09:45:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [9c00753] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7a92319 >2013-03-08 09:45:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [9c00753] FINISH, GetHardwareInfoVDSCommand, log id: 7a92319 >2013-03-08 09:45:00,258 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-11) [34d0841c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:45:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-11) [2c9c79ad] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:45:00,262 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-11) [2c9c79ad] FINISH, ActivateVdsVDSCommand, log id: 237da3f1 >2013-03-08 09:45:00,281 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-11) [2c9c79ad] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:45:00,282 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-11) [2c9c79ad] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 09:45:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) [2c9c79ad] Autorecovering 1 storage domains >2013-03-08 09:45:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) [2c9c79ad] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 09:45:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-11) [2ab50aab] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:45:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-11) [2ab50aab] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 9:45 AM >2013-03-08 09:45:00,287 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-11) [2ab50aab] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 9:45 AM >2013-03-08 09:45:01,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [44062586] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: fbae99 >2013-03-08 09:45:01,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [44062586] FINISH, GetHardwareInfoVDSCommand, log id: fbae99 >2013-03-08 09:45:01,061 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-80) [71d02070] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 09:45:01,065 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-80) [5b8e012b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 09:45:01,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7494fac >2013-03-08 09:46:06,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 09:46:06,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:46:06,071 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 09:46:06,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 7494fac >2013-03-08 09:46:06,075 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [7a3c1fdc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:46:06,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [7a3c1fdc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 309c67a9 >2013-03-08 09:46:06,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [7a3c1fdc] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 09:46:06,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [7a3c1fdc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 309c67a9 >2013-03-08 09:46:06,092 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [7a3c1fdc] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 09:46:06,093 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [5b8e012b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 09:46:06,105 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-80) [344baf35] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:46:06,106 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [344baf35] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1f95d944 >2013-03-08 09:46:06,108 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [344baf35] FINISH, SetVdsStatusVDSCommand, log id: 1f95d944 >2013-03-08 09:46:06,113 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-80) [344baf35] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 09:46:06,140 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-80) [3fdc3cf8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:46:06,142 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-80) [6399daff] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:46:06,143 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-80) [6399daff] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 09:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [53830029] Autorecovering 1 hosts >2013-03-08 09:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [53830029] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [594d8365] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [594d8365] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [594d8365] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3623af02 >2013-03-08 09:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [594d8365] FINISH, SetVdsStatusVDSCommand, log id: 3623af02 >2013-03-08 09:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-87) [594d8365] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 75c671b0 >2013-03-08 09:50:00,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [594d8365] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4fa89d44 >2013-03-08 09:50:00,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [594d8365] FINISH, GetHardwareInfoVDSCommand, log id: 4fa89d44 >2013-03-08 09:50:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [3abd93b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:50:00,262 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [5785c2e1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:50:00,263 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-87) [5785c2e1] FINISH, ActivateVdsVDSCommand, log id: 75c671b0 >2013-03-08 09:50:00,283 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [5785c2e1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:50:00,284 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [5785c2e1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 09:50:00,285 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [5785c2e1] Autorecovering 1 storage domains >2013-03-08 09:50:00,285 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [5785c2e1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 09:50:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [515db1bc] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:50:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [515db1bc] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 9:50 AM >2013-03-08 09:50:00,289 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [515db1bc] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 9:50 AM >2013-03-08 09:50:00,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [5d0ef834] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 87ec4b3 >2013-03-08 09:50:00,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [5d0ef834] FINISH, GetHardwareInfoVDSCommand, log id: 87ec4b3 >2013-03-08 09:50:00,384 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-17) [7b94f28f] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 09:50:00,388 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-17) [6116359b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 09:50:00,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2c5038ea >2013-03-08 09:51:05,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 09:51:05,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:51:05,356 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 09:51:05,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 2c5038ea >2013-03-08 09:51:05,360 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [60243fb9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:51:05,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [60243fb9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3824f930 >2013-03-08 09:51:05,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [60243fb9] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 09:51:05,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [60243fb9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3824f930 >2013-03-08 09:51:05,376 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [60243fb9] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 09:51:05,377 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [6116359b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 09:51:05,386 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-17) [3cf7ca35] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:51:05,388 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [3cf7ca35] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: ecdcfe5 >2013-03-08 09:51:05,389 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [3cf7ca35] FINISH, SetVdsStatusVDSCommand, log id: ecdcfe5 >2013-03-08 09:51:05,395 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [3cf7ca35] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 09:51:05,416 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [1f0c6ca1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:51:05,418 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [7827ffd1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:51:05,419 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-17) [7827ffd1] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 09:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [c9ee581] Autorecovering 1 hosts >2013-03-08 09:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [c9ee581] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-49) [157581a7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-49) [157581a7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [157581a7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 552b9344 >2013-03-08 09:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [157581a7] FINISH, SetVdsStatusVDSCommand, log id: 552b9344 >2013-03-08 09:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-49) [157581a7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 624acf83 >2013-03-08 09:55:00,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [157581a7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6f0bea68 >2013-03-08 09:55:00,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [157581a7] FINISH, GetHardwareInfoVDSCommand, log id: 6f0bea68 >2013-03-08 09:55:00,409 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-49) [77590a78] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:55:00,411 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-49) [4441a2e8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:55:00,412 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-49) [4441a2e8] FINISH, ActivateVdsVDSCommand, log id: 624acf83 >2013-03-08 09:55:00,432 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-49) [4441a2e8] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 09:55:00,433 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-49) [4441a2e8] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 09:55:00,434 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [4441a2e8] Autorecovering 1 storage domains >2013-03-08 09:55:00,434 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [4441a2e8] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 09:55:00,435 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-49) [171848f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:55:00,435 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-49) [171848f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 9:55 AM >2013-03-08 09:55:00,437 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-49) [171848f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 9:55 AM >2013-03-08 09:55:02,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [47a66593] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 15e612f7 >2013-03-08 09:55:02,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [47a66593] FINISH, GetHardwareInfoVDSCommand, log id: 15e612f7 >2013-03-08 09:55:02,663 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-9) [3f299de6] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 09:55:02,668 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-9) [7b8ac070] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 09:55:02,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 62fe24ba >2013-03-08 09:56:09,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 09:56:09,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 09:56:09,716 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 09:56:09,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 62fe24ba >2013-03-08 09:56:09,720 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [635047b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 09:56:09,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [635047b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5eda519e >2013-03-08 09:56:09,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [635047b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 09:56:09,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [635047b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5eda519e >2013-03-08 09:56:09,736 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [635047b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 09:56:09,736 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [7b8ac070] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 09:56:09,746 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-9) [1dd9ee28] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:56:09,747 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [1dd9ee28] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: fdb86d2 >2013-03-08 09:56:09,749 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [1dd9ee28] FINISH, SetVdsStatusVDSCommand, log id: fdb86d2 >2013-03-08 09:56:09,754 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-9) [1dd9ee28] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 09:56:09,775 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-9) [a23615f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:56:09,777 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-9) [452d305b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 09:56:09,778 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-9) [452d305b] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 10:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [5404b7e7] Autorecovering 1 hosts >2013-03-08 10:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [5404b7e7] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [1e467ff9] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [1e467ff9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [1e467ff9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 55bd6cee >2013-03-08 10:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [1e467ff9] FINISH, SetVdsStatusVDSCommand, log id: 55bd6cee >2013-03-08 10:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-47) [1e467ff9] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2bf0f56e >2013-03-08 10:00:00,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [1e467ff9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 501740a4 >2013-03-08 10:00:00,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [1e467ff9] FINISH, GetHardwareInfoVDSCommand, log id: 501740a4 >2013-03-08 10:00:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [cd30753] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:00:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [25be25e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:00:00,206 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-47) [25be25e] FINISH, ActivateVdsVDSCommand, log id: 2bf0f56e >2013-03-08 10:00:00,225 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [25be25e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:00:00,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [25be25e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 10:00:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [25be25e] Autorecovering 1 storage domains >2013-03-08 10:00:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [25be25e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 10:00:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [66046291] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:00:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [66046291] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 10:00 AM >2013-03-08 10:00:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [66046291] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 10:00 AM >2013-03-08 10:00:00,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [535ce846] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4d9706a2 >2013-03-08 10:00:00,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [535ce846] FINISH, GetHardwareInfoVDSCommand, log id: 4d9706a2 >2013-03-08 10:00:01,021 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-74) [1e3146b] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 10:00:01,024 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-74) [3cf5a779] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 10:00:01,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 639ec909 >2013-03-08 10:01:05,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 10:01:05,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:01:05,025 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 10:01:05,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 639ec909 >2013-03-08 10:01:05,029 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [63278b5a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:01:05,043 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [63278b5a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 63b561f2 >2013-03-08 10:01:05,043 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [63278b5a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 10:01:05,043 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [63278b5a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 63b561f2 >2013-03-08 10:01:05,045 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [63278b5a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 10:01:05,045 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [3cf5a779] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 10:01:05,062 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-74) [18f3f4f1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:01:05,063 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [18f3f4f1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 28465f8e >2013-03-08 10:01:05,064 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [18f3f4f1] FINISH, SetVdsStatusVDSCommand, log id: 28465f8e >2013-03-08 10:01:05,071 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-74) [18f3f4f1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 10:01:05,091 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-74) [25ece554] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:01:05,093 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-74) [1e53741e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:01:05,094 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-74) [1e53741e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 10:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-71) [8126383] Autorecovering 1 hosts >2013-03-08 10:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-71) [8126383] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-71) [32145e54] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-71) [32145e54] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-71) [32145e54] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 223fd636 >2013-03-08 10:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-71) [32145e54] FINISH, SetVdsStatusVDSCommand, log id: 223fd636 >2013-03-08 10:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-71) [32145e54] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 741c5621 >2013-03-08 10:05:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-71) [32145e54] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: cff2029 >2013-03-08 10:05:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-71) [32145e54] FINISH, GetHardwareInfoVDSCommand, log id: cff2029 >2013-03-08 10:05:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-71) [3554e294] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:05:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-71) [4ffc92a0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:05:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-71) [4ffc92a0] FINISH, ActivateVdsVDSCommand, log id: 741c5621 >2013-03-08 10:05:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-71) [4ffc92a0] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:05:00,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-71) [4ffc92a0] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 10:05:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-71) [4ffc92a0] Autorecovering 1 storage domains >2013-03-08 10:05:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-71) [4ffc92a0] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 10:05:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-71) [57ef1896] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:05:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-71) [57ef1896] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 10:05 AM >2013-03-08 10:05:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-71) [57ef1896] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 10:05 AM >2013-03-08 10:05:02,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [75293efe] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5e33d8d4 >2013-03-08 10:05:02,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [75293efe] FINISH, GetHardwareInfoVDSCommand, log id: 5e33d8d4 >2013-03-08 10:05:02,365 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-32) [4b847c85] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 10:05:02,368 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-32) [4e69cd26] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 10:05:02,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 50a5f743 >2013-03-08 10:06:08,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 10:06:08,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:06:08,368 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 10:06:08,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 50a5f743 >2013-03-08 10:06:08,372 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [5bba3114] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:06:08,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5bba3114] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6d5562c4 >2013-03-08 10:06:08,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [5bba3114] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 10:06:08,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5bba3114] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6d5562c4 >2013-03-08 10:06:08,388 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [5bba3114] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 10:06:08,388 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [4e69cd26] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 10:06:08,398 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-32) [47c47c06] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:06:08,403 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [47c47c06] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4c6d072a >2013-03-08 10:06:08,404 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [47c47c06] FINISH, SetVdsStatusVDSCommand, log id: 4c6d072a >2013-03-08 10:06:08,409 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-32) [47c47c06] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 10:06:08,432 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-32) [445993cb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:06:08,434 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-32) [525c12a4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:06:08,435 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-32) [525c12a4] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 10:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [31135343] Autorecovering 1 hosts >2013-03-08 10:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [31135343] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [971e896] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [971e896] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [971e896] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 193e6a03 >2013-03-08 10:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [971e896] FINISH, SetVdsStatusVDSCommand, log id: 193e6a03 >2013-03-08 10:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [971e896] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 23dc302f >2013-03-08 10:10:00,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [971e896] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 19efcc07 >2013-03-08 10:10:00,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [971e896] FINISH, GetHardwareInfoVDSCommand, log id: 19efcc07 >2013-03-08 10:10:00,533 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-45) [3f0d4c34] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:10:00,535 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-45) [1b195ad6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:10:00,536 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [1b195ad6] FINISH, ActivateVdsVDSCommand, log id: 23dc302f >2013-03-08 10:10:00,556 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [1b195ad6] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:10:00,557 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-45) [1b195ad6] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 10:10:00,558 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [1b195ad6] Autorecovering 1 storage domains >2013-03-08 10:10:00,558 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [1b195ad6] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 10:10:00,559 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [19d932c5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:10:00,559 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [19d932c5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 10:10 AM >2013-03-08 10:10:00,562 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [19d932c5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 10:10 AM >2013-03-08 10:10:02,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [470c8030] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2aa09aa0 >2013-03-08 10:10:02,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [470c8030] FINISH, GetHardwareInfoVDSCommand, log id: 2aa09aa0 >2013-03-08 10:10:02,678 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-53) [9acfd37] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 10:10:02,681 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-53) [b5c7acd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 10:10:02,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6f9b5bab >2013-03-08 10:11:06,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 10:11:06,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:11:06,718 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 10:11:06,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 6f9b5bab >2013-03-08 10:11:06,722 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [4a229439] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:11:06,736 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [4a229439] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a411655 >2013-03-08 10:11:06,736 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [4a229439] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 10:11:06,736 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [4a229439] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a411655 >2013-03-08 10:11:06,738 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [4a229439] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 10:11:06,738 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [b5c7acd] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 10:11:06,749 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-53) [7a58f6ee] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:11:06,750 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [7a58f6ee] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 35155612 >2013-03-08 10:11:06,751 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [7a58f6ee] FINISH, SetVdsStatusVDSCommand, log id: 35155612 >2013-03-08 10:11:06,758 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-53) [7a58f6ee] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 10:11:06,791 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-53) [6b73d759] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:11:06,793 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-53) [2694c156] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:11:06,795 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-53) [2694c156] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 10:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [6f15c46e] Autorecovering 1 hosts >2013-03-08 10:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [6f15c46e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [316834ce] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:15:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [316834ce] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [316834ce] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 53bdfcd6 >2013-03-08 10:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [316834ce] FINISH, SetVdsStatusVDSCommand, log id: 53bdfcd6 >2013-03-08 10:15:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [316834ce] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 30bcce23 >2013-03-08 10:15:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [316834ce] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 495a4c0a >2013-03-08 10:15:00,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [316834ce] FINISH, GetHardwareInfoVDSCommand, log id: 495a4c0a >2013-03-08 10:15:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [587165ad] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:15:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [45ad8502] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:15:00,093 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [45ad8502] FINISH, ActivateVdsVDSCommand, log id: 30bcce23 >2013-03-08 10:15:00,112 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [45ad8502] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:15:00,113 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [45ad8502] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 10:15:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [45ad8502] Autorecovering 1 storage domains >2013-03-08 10:15:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [45ad8502] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 10:15:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [423ecb73] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:15:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [423ecb73] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 10:15 AM >2013-03-08 10:15:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [423ecb73] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 10:15 AM >2013-03-08 10:15:00,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [4d747221] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7c78a2ac >2013-03-08 10:15:01,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [4d747221] FINISH, GetHardwareInfoVDSCommand, log id: 7c78a2ac >2013-03-08 10:15:01,042 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-7) [2c14b11c] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 10:15:01,045 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-7) [2a46bf7c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 10:15:01,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 46feb3f0 >2013-03-08 10:16:06,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 10:16:06,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:16:06,036 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 10:16:06,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 46feb3f0 >2013-03-08 10:16:06,040 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [40727395] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:16:06,054 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [40727395] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 62dde65b >2013-03-08 10:16:06,054 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [40727395] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 10:16:06,055 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [40727395] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 62dde65b >2013-03-08 10:16:06,057 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [40727395] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 10:16:06,057 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [2a46bf7c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 10:16:06,067 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-7) [144618e6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:16:06,069 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [144618e6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 64c0b9ea >2013-03-08 10:16:06,070 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [144618e6] FINISH, SetVdsStatusVDSCommand, log id: 64c0b9ea >2013-03-08 10:16:06,077 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-7) [144618e6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 10:16:06,098 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-7) [70a6b87f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:16:06,101 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-7) [463f5475] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:16:06,102 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) [463f5475] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 10:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [2219fb2] Autorecovering 1 hosts >2013-03-08 10:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [2219fb2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-10) [3c8a7082] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-10) [3c8a7082] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [3c8a7082] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2196d74a >2013-03-08 10:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [3c8a7082] FINISH, SetVdsStatusVDSCommand, log id: 2196d74a >2013-03-08 10:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-10) [3c8a7082] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: f236a4d >2013-03-08 10:20:00,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [3c8a7082] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 76a45084 >2013-03-08 10:20:00,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [3c8a7082] FINISH, GetHardwareInfoVDSCommand, log id: 76a45084 >2013-03-08 10:20:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-10) [443825d4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:20:00,263 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-10) [6699ccfb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:20:00,264 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-10) [6699ccfb] FINISH, ActivateVdsVDSCommand, log id: f236a4d >2013-03-08 10:20:00,284 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-10) [6699ccfb] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:20:00,284 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-10) [6699ccfb] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 10:20:00,286 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [6699ccfb] Autorecovering 1 storage domains >2013-03-08 10:20:00,286 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [6699ccfb] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 10:20:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-10) [c72cc82] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:20:00,287 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-10) [c72cc82] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 10:20 AM >2013-03-08 10:20:00,289 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-10) [c72cc82] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 10:20 AM >2013-03-08 10:20:00,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [2cb65d8f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 44fd1938 >2013-03-08 10:20:00,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [2cb65d8f] FINISH, GetHardwareInfoVDSCommand, log id: 44fd1938 >2013-03-08 10:20:00,362 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-72) [4448ee09] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 10:20:00,365 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-72) [419b5caa] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 10:20:00,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 98b9f33 >2013-03-08 10:21:05,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 10:21:05,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:21:05,361 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 10:21:05,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 98b9f33 >2013-03-08 10:21:05,365 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [6a846227] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:21:05,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [6a846227] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 44b61fa7 >2013-03-08 10:21:05,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [6a846227] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 10:21:05,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [6a846227] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 44b61fa7 >2013-03-08 10:21:05,381 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [6a846227] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 10:21:05,381 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [419b5caa] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 10:21:05,392 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-72) [6170263d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:21:05,393 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [6170263d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 18729300 >2013-03-08 10:21:05,394 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [6170263d] FINISH, SetVdsStatusVDSCommand, log id: 18729300 >2013-03-08 10:21:05,401 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [6170263d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 10:21:05,437 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [521d4c3e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:21:05,440 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [4a6737d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:21:05,441 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-72) [4a6737d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 10:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [5c8aad20] Autorecovering 1 hosts >2013-03-08 10:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [5c8aad20] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [3629ae4e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [3629ae4e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [3629ae4e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3ed3dc75 >2013-03-08 10:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [3629ae4e] FINISH, SetVdsStatusVDSCommand, log id: 3ed3dc75 >2013-03-08 10:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [3629ae4e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3b17e749 >2013-03-08 10:25:00,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [3629ae4e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1f5cd374 >2013-03-08 10:25:00,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [3629ae4e] FINISH, GetHardwareInfoVDSCommand, log id: 1f5cd374 >2013-03-08 10:25:00,297 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-16) [a270f33] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:25:00,299 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-16) [3f50e50a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:25:00,301 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [3f50e50a] FINISH, ActivateVdsVDSCommand, log id: 3b17e749 >2013-03-08 10:25:00,320 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [3f50e50a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:25:00,321 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-16) [3f50e50a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 10:25:00,322 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [3f50e50a] Autorecovering 1 storage domains >2013-03-08 10:25:00,322 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [3f50e50a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 10:25:00,323 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [6e14840e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:25:00,323 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [6e14840e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 10:25 AM >2013-03-08 10:25:00,326 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [6e14840e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 10:25 AM >2013-03-08 10:25:02,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [51204c0f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 16a4c232 >2013-03-08 10:25:02,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [51204c0f] FINISH, GetHardwareInfoVDSCommand, log id: 16a4c232 >2013-03-08 10:25:02,681 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-90) [71a3826d] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 10:25:02,685 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-90) [e95d417] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 10:25:02,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 19b6b7d6 >2013-03-08 10:26:08,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 10:26:08,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:26:08,730 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 10:26:08,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 19b6b7d6 >2013-03-08 10:26:08,735 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [711ed9cb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:26:08,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [711ed9cb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 24cea87 >2013-03-08 10:26:08,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [711ed9cb] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 10:26:08,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [711ed9cb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 24cea87 >2013-03-08 10:26:08,751 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [711ed9cb] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 10:26:08,752 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [e95d417] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 10:26:08,767 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-90) [4e0aff61] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:26:08,768 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [4e0aff61] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6e0799ee >2013-03-08 10:26:08,769 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [4e0aff61] FINISH, SetVdsStatusVDSCommand, log id: 6e0799ee >2013-03-08 10:26:08,774 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [4e0aff61] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 10:26:08,800 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-90) [126b2823] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:26:08,803 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-90) [78204549] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:26:08,804 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-90) [78204549] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 10:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [356167d] Autorecovering 1 hosts >2013-03-08 10:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [356167d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [5c5b2f2c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [5c5b2f2c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [5c5b2f2c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: e40eac2 >2013-03-08 10:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [5c5b2f2c] FINISH, SetVdsStatusVDSCommand, log id: e40eac2 >2013-03-08 10:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [5c5b2f2c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 31579ecf >2013-03-08 10:30:00,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [5c5b2f2c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2ee1e22c >2013-03-08 10:30:00,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [5c5b2f2c] FINISH, GetHardwareInfoVDSCommand, log id: 2ee1e22c >2013-03-08 10:30:00,409 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-69) [1a20bd56] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:30:00,412 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-69) [757703ac] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:30:00,413 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [757703ac] FINISH, ActivateVdsVDSCommand, log id: 31579ecf >2013-03-08 10:30:00,433 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [757703ac] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:30:00,434 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-69) [757703ac] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 10:30:00,435 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [757703ac] Autorecovering 1 storage domains >2013-03-08 10:30:00,435 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [757703ac] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 10:30:00,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [375b599c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:30:00,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [375b599c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 10:30 AM >2013-03-08 10:30:00,439 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [375b599c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 10:30 AM >2013-03-08 10:30:02,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [1d77d63b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 26662a5a >2013-03-08 10:30:03,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [1d77d63b] FINISH, GetHardwareInfoVDSCommand, log id: 26662a5a >2013-03-08 10:30:03,047 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-96) [17524e7e] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 10:30:03,050 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-96) [9139211] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 10:30:03,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 21979737 >2013-03-08 10:30:33,736 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-51) [75c8a919] Start refreshing all users data >2013-03-08 10:30:34,676 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-67) [6ffac8dc] Attempting to update VMs/Templates Ovf. >2013-03-08 10:30:34,677 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-67) [6ffac8dc] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 10:30:34,678 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-67) [6ffac8dc] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 10:30:34,678 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-67) [6ffac8dc] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 10:30:34,678 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-67) [6ffac8dc] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 10:30:34,678 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-67) [6ffac8dc] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 10:30:34,679 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-67) [6ffac8dc] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 10:31:08,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 10:31:08,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:31:08,062 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 10:31:08,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 21979737 >2013-03-08 10:31:08,066 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [15acdf30] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:31:08,080 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [15acdf30] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 577a30a2 >2013-03-08 10:31:08,080 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [15acdf30] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 10:31:08,080 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [15acdf30] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 577a30a2 >2013-03-08 10:31:08,082 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [15acdf30] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 10:31:08,082 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [9139211] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 10:31:08,092 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-96) [9667fdc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:31:08,094 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [9667fdc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 130509b7 >2013-03-08 10:31:08,095 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [9667fdc] FINISH, SetVdsStatusVDSCommand, log id: 130509b7 >2013-03-08 10:31:08,101 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-96) [9667fdc] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 10:31:08,122 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-96) [1acd1326] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:31:08,124 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-96) [539dc993] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:31:08,126 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-96) [539dc993] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 10:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [6097ad7] Autorecovering 1 hosts >2013-03-08 10:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [6097ad7] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [7cebca8f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [7cebca8f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [7cebca8f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3e0eca37 >2013-03-08 10:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [7cebca8f] FINISH, SetVdsStatusVDSCommand, log id: 3e0eca37 >2013-03-08 10:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [7cebca8f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 57762296 >2013-03-08 10:35:00,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [7cebca8f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 34a86d04 >2013-03-08 10:35:00,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [7cebca8f] FINISH, GetHardwareInfoVDSCommand, log id: 34a86d04 >2013-03-08 10:35:00,186 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [ccef2f3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:35:00,188 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [2ff77289] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:35:00,189 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [2ff77289] FINISH, ActivateVdsVDSCommand, log id: 57762296 >2013-03-08 10:35:00,209 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [2ff77289] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:35:00,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [2ff77289] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 10:35:00,211 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [2ff77289] Autorecovering 1 storage domains >2013-03-08 10:35:00,211 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [2ff77289] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 10:35:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [2beb7743] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:35:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [2beb7743] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 10:35 AM >2013-03-08 10:35:00,214 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [2beb7743] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 10:35 AM >2013-03-08 10:35:02,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [1ca1e668] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3ca3bb6d >2013-03-08 10:35:02,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [1ca1e668] FINISH, GetHardwareInfoVDSCommand, log id: 3ca3bb6d >2013-03-08 10:35:02,374 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-13) [42149381] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 10:35:02,377 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-13) [2d4eb19b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 10:35:02,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1158caf4 >2013-03-08 10:36:07,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 10:36:07,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:36:07,358 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 10:36:07,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 1158caf4 >2013-03-08 10:36:07,362 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [4f9ac66c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:36:07,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [4f9ac66c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 681b0ea9 >2013-03-08 10:36:07,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [4f9ac66c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 10:36:07,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [4f9ac66c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 681b0ea9 >2013-03-08 10:36:07,378 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [4f9ac66c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 10:36:07,379 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [2d4eb19b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 10:36:07,388 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-13) [5f2171d6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:36:07,390 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [5f2171d6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: fb0226b >2013-03-08 10:36:07,391 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [5f2171d6] FINISH, SetVdsStatusVDSCommand, log id: fb0226b >2013-03-08 10:36:07,397 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-13) [5f2171d6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 10:36:07,417 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-13) [2304a9d5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:36:07,419 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-13) [7e57290] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:36:07,421 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-13) [7e57290] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 10:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [6892803f] Autorecovering 1 hosts >2013-03-08 10:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [6892803f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [12b6c3fe] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [12b6c3fe] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [12b6c3fe] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 67d519c2 >2013-03-08 10:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [12b6c3fe] FINISH, SetVdsStatusVDSCommand, log id: 67d519c2 >2013-03-08 10:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-77) [12b6c3fe] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 31fcd66c >2013-03-08 10:40:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [12b6c3fe] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7202e5ff >2013-03-08 10:40:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [12b6c3fe] FINISH, GetHardwareInfoVDSCommand, log id: 7202e5ff >2013-03-08 10:40:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-77) [72ec50b6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:40:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-77) [257f6ed4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:40:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-77) [257f6ed4] FINISH, ActivateVdsVDSCommand, log id: 31fcd66c >2013-03-08 10:40:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [257f6ed4] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:40:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-77) [257f6ed4] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 10:40:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [257f6ed4] Autorecovering 1 storage domains >2013-03-08 10:40:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [257f6ed4] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 10:40:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [1e8ba581] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:40:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [1e8ba581] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 10:40 AM >2013-03-08 10:40:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [1e8ba581] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 10:40 AM >2013-03-08 10:40:01,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [20224cd5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6b824da0 >2013-03-08 10:40:01,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [20224cd5] FINISH, GetHardwareInfoVDSCommand, log id: 6b824da0 >2013-03-08 10:40:01,661 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-60) [7eaa5f3f] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 10:40:01,664 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-60) [49d65cc8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 10:40:01,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2ba595c5 >2013-03-08 10:41:06,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 10:41:06,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:41:06,665 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 10:41:06,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 2ba595c5 >2013-03-08 10:41:06,669 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [d14b6de] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:41:06,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [d14b6de] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 641ede75 >2013-03-08 10:41:06,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [d14b6de] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 10:41:06,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [d14b6de] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 641ede75 >2013-03-08 10:41:06,685 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [d14b6de] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 10:41:06,685 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [49d65cc8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 10:41:06,695 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-60) [5e148291] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:41:06,696 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [5e148291] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 45d238fb >2013-03-08 10:41:06,698 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [5e148291] FINISH, SetVdsStatusVDSCommand, log id: 45d238fb >2013-03-08 10:41:06,704 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-60) [5e148291] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 10:41:06,725 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-60) [959e90a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:41:06,728 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-60) [28f6d5ca] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:41:06,729 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-60) [28f6d5ca] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 10:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [f72379c] Autorecovering 1 hosts >2013-03-08 10:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [f72379c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [1955dbfe] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [1955dbfe] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [1955dbfe] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2845e94c >2013-03-08 10:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [1955dbfe] FINISH, SetVdsStatusVDSCommand, log id: 2845e94c >2013-03-08 10:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-89) [1955dbfe] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7bb4318d >2013-03-08 10:45:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [1955dbfe] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2306d344 >2013-03-08 10:45:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [1955dbfe] FINISH, GetHardwareInfoVDSCommand, log id: 2306d344 >2013-03-08 10:45:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-89) [74701078] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:45:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-89) [3463afaa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:45:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-89) [3463afaa] FINISH, ActivateVdsVDSCommand, log id: 7bb4318d >2013-03-08 10:45:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [3463afaa] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:45:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-89) [3463afaa] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 10:45:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [3463afaa] Autorecovering 1 storage domains >2013-03-08 10:45:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [3463afaa] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 10:45:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [3f0da4d7] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:45:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [3f0da4d7] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 10:45 AM >2013-03-08 10:45:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [3f0da4d7] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 10:45 AM >2013-03-08 10:45:00,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [55006d3f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 69ed6272 >2013-03-08 10:45:00,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [55006d3f] FINISH, GetHardwareInfoVDSCommand, log id: 69ed6272 >2013-03-08 10:45:00,990 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-22) [550921b] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 10:45:00,993 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-22) [7913b1de] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 10:45:01,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4a2c5a6d >2013-03-08 10:46:04,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 10:46:04,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:46:04,984 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 10:46:04,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 4a2c5a6d >2013-03-08 10:46:04,988 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [56910b1d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:46:05,002 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [56910b1d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5fa54847 >2013-03-08 10:46:05,002 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [56910b1d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 10:46:05,002 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [56910b1d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5fa54847 >2013-03-08 10:46:05,007 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [7913b1de] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 10:46:05,016 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-22) [599e9cf7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:46:05,018 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [599e9cf7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4081d7a5 >2013-03-08 10:46:05,019 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [599e9cf7] FINISH, SetVdsStatusVDSCommand, log id: 4081d7a5 >2013-03-08 10:46:05,026 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [599e9cf7] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 10:46:05,046 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [f50837e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:46:05,048 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [59973c55] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:46:05,049 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-22) [59973c55] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 10:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) Autorecovering 1 hosts >2013-03-08 10:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-55) [26cdeae4] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-55) [26cdeae4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [26cdeae4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1b7d1abc >2013-03-08 10:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [26cdeae4] FINISH, SetVdsStatusVDSCommand, log id: 1b7d1abc >2013-03-08 10:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-55) [26cdeae4] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2b8c5c5e >2013-03-08 10:50:00,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [26cdeae4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3cee5014 >2013-03-08 10:50:00,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [26cdeae4] FINISH, GetHardwareInfoVDSCommand, log id: 3cee5014 >2013-03-08 10:50:00,187 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-55) [51fddaee] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:50:00,190 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-55) [d752e94] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:50:00,191 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-55) [d752e94] FINISH, ActivateVdsVDSCommand, log id: 2b8c5c5e >2013-03-08 10:50:00,210 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-55) [d752e94] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:50:00,211 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-55) [d752e94] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 10:50:00,212 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [d752e94] Autorecovering 1 storage domains >2013-03-08 10:50:00,212 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [d752e94] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 10:50:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-55) [6046fd1c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:50:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-55) [6046fd1c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 10:50 AM >2013-03-08 10:50:00,216 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-55) [6046fd1c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 10:50 AM >2013-03-08 10:50:02,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [33ae847e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 203638 >2013-03-08 10:50:02,267 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [33ae847e] FINISH, GetHardwareInfoVDSCommand, log id: 203638 >2013-03-08 10:50:02,305 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-31) [503ba927] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 10:50:02,308 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-31) [28e8155a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 10:50:02,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 20c0bfad >2013-03-08 10:51:07,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 10:51:07,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:51:07,326 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 10:51:07,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 20c0bfad >2013-03-08 10:51:07,330 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [28f5caaa] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:51:07,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [28f5caaa] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1790e2b1 >2013-03-08 10:51:07,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [28f5caaa] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 10:51:07,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [28f5caaa] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1790e2b1 >2013-03-08 10:51:07,346 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [28f5caaa] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 10:51:07,346 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [28e8155a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 10:51:07,356 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-31) [111b84f8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:51:07,358 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [111b84f8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 19d97c2e >2013-03-08 10:51:07,359 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [111b84f8] FINISH, SetVdsStatusVDSCommand, log id: 19d97c2e >2013-03-08 10:51:07,365 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-31) [111b84f8] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 10:51:07,386 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-31) [5922e4e7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:51:07,388 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-31) [290d4943] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:51:07,390 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-31) [290d4943] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 10:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [621c65a2] Autorecovering 1 hosts >2013-03-08 10:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [621c65a2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [406650f9] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [406650f9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [406650f9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 14c47f20 >2013-03-08 10:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [406650f9] FINISH, SetVdsStatusVDSCommand, log id: 14c47f20 >2013-03-08 10:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-70) [406650f9] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 316f4816 >2013-03-08 10:55:00,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [406650f9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6962c8c9 >2013-03-08 10:55:00,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [406650f9] FINISH, GetHardwareInfoVDSCommand, log id: 6962c8c9 >2013-03-08 10:55:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-70) [14264da9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:55:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-70) [47503391] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:55:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-70) [47503391] FINISH, ActivateVdsVDSCommand, log id: 316f4816 >2013-03-08 10:55:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [47503391] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 10:55:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-70) [47503391] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 10:55:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [47503391] Autorecovering 1 storage domains >2013-03-08 10:55:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [47503391] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 10:55:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [4390d057] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:55:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [4390d057] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 10:55 AM >2013-03-08 10:55:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [4390d057] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 10:55 AM >2013-03-08 10:55:01,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [2cbaa5b5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2f1bcc8c >2013-03-08 10:55:01,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [2cbaa5b5] FINISH, GetHardwareInfoVDSCommand, log id: 2f1bcc8c >2013-03-08 10:55:01,640 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-78) [299f8e31] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 10:55:01,643 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-78) [3ba87480] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 10:55:01,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7a3485c5 >2013-03-08 10:56:05,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 10:56:05,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 10:56:05,590 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 10:56:05,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 7a3485c5 >2013-03-08 10:56:05,594 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [39c6c089] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 10:56:05,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [39c6c089] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3d0a93bb >2013-03-08 10:56:05,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [39c6c089] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 10:56:05,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [39c6c089] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3d0a93bb >2013-03-08 10:56:05,610 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [39c6c089] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 10:56:05,611 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [3ba87480] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 10:56:05,620 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-78) [11c08c7e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:56:05,622 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [11c08c7e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6181a785 >2013-03-08 10:56:05,623 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [11c08c7e] FINISH, SetVdsStatusVDSCommand, log id: 6181a785 >2013-03-08 10:56:05,629 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-78) [11c08c7e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 10:56:05,650 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-78) [5c570cd7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:56:05,653 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-78) [dfb5ca5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 10:56:05,654 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-78) [dfb5ca5] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 11:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [2d106f9d] Autorecovering 1 hosts >2013-03-08 11:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [2d106f9d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-12) [284c245e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-12) [284c245e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [284c245e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 26afb042 >2013-03-08 11:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [284c245e] FINISH, SetVdsStatusVDSCommand, log id: 26afb042 >2013-03-08 11:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-12) [284c245e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 78f00ca5 >2013-03-08 11:00:00,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [284c245e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 75e01ff3 >2013-03-08 11:00:00,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [284c245e] FINISH, GetHardwareInfoVDSCommand, log id: 75e01ff3 >2013-03-08 11:00:00,294 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-12) [4d4e2159] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:00:00,296 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-12) [318b25a7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:00:00,298 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-12) [318b25a7] FINISH, ActivateVdsVDSCommand, log id: 78f00ca5 >2013-03-08 11:00:00,318 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-12) [318b25a7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:00:00,319 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-12) [318b25a7] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 11:00:00,320 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [318b25a7] Autorecovering 1 storage domains >2013-03-08 11:00:00,320 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [318b25a7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 11:00:00,321 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-12) [dd0195b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:00:00,321 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-12) [dd0195b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 11:00 AM >2013-03-08 11:00:00,324 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-12) [dd0195b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 11:00 AM >2013-03-08 11:00:02,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [5c67c943] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 715e968f >2013-03-08 11:00:02,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [5c67c943] FINISH, GetHardwareInfoVDSCommand, log id: 715e968f >2013-03-08 11:00:02,899 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-83) [2da1273a] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 11:00:02,902 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-83) [140f575d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 11:00:02,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 45138adf >2013-03-08 11:01:08,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 11:01:08,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:01:08,927 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 11:01:08,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 45138adf >2013-03-08 11:01:08,931 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [5c9f743a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:01:08,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5c9f743a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 363e6a30 >2013-03-08 11:01:08,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [5c9f743a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 11:01:08,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5c9f743a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 363e6a30 >2013-03-08 11:01:08,947 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [5c9f743a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 11:01:08,947 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [140f575d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 11:01:08,956 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-83) [294b6648] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:01:08,958 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [294b6648] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4474b233 >2013-03-08 11:01:08,959 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [294b6648] FINISH, SetVdsStatusVDSCommand, log id: 4474b233 >2013-03-08 11:01:08,965 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-83) [294b6648] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 11:01:08,986 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-83) [693d7600] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:01:08,988 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-83) [1a249fa6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:01:08,989 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-83) [1a249fa6] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 11:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [525c12a4] Autorecovering 1 hosts >2013-03-08 11:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [525c12a4] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [7abadc55] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [7abadc55] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [7abadc55] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 43a12118 >2013-03-08 11:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [7abadc55] FINISH, SetVdsStatusVDSCommand, log id: 43a12118 >2013-03-08 11:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-32) [7abadc55] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4bd71a3e >2013-03-08 11:05:00,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [7abadc55] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 38f6eb58 >2013-03-08 11:05:00,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [7abadc55] FINISH, GetHardwareInfoVDSCommand, log id: 38f6eb58 >2013-03-08 11:05:00,240 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-32) [3df021db] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:05:00,243 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-32) [530d6fcb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:05:00,245 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-32) [530d6fcb] FINISH, ActivateVdsVDSCommand, log id: 4bd71a3e >2013-03-08 11:05:00,264 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [530d6fcb] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:05:00,265 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-32) [530d6fcb] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 11:05:00,266 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [530d6fcb] Autorecovering 1 storage domains >2013-03-08 11:05:00,266 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [530d6fcb] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 11:05:00,267 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [4a6c8eaf] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:05:00,267 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [4a6c8eaf] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 11:05 AM >2013-03-08 11:05:00,270 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [4a6c8eaf] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 11:05 AM >2013-03-08 11:05:00,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [75c8a919] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 228819ff >2013-03-08 11:05:00,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [75c8a919] FINISH, GetHardwareInfoVDSCommand, log id: 228819ff >2013-03-08 11:05:00,358 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-51) [653dd17c] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 11:05:00,362 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-51) [2e821120] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 11:05:00,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 390d3a9 >2013-03-08 11:06:05,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 11:06:05,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:06:05,357 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 11:06:05,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 390d3a9 >2013-03-08 11:06:05,361 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [461f925a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:06:05,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [461f925a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 781edd45 >2013-03-08 11:06:05,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [461f925a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 11:06:05,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [461f925a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 781edd45 >2013-03-08 11:06:05,377 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [461f925a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 11:06:05,378 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [2e821120] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 11:06:05,387 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-51) [70faee43] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:06:05,389 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [70faee43] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2aedd368 >2013-03-08 11:06:05,390 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [70faee43] FINISH, SetVdsStatusVDSCommand, log id: 2aedd368 >2013-03-08 11:06:05,396 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-51) [70faee43] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 11:06:05,417 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-51) [56995065] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:06:05,419 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-51) [728b1608] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:06:05,420 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-51) [728b1608] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 11:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [2694c156] Autorecovering 1 hosts >2013-03-08 11:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [2694c156] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-53) [4960f98a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-53) [4960f98a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [4960f98a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5a97032e >2013-03-08 11:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [4960f98a] FINISH, SetVdsStatusVDSCommand, log id: 5a97032e >2013-03-08 11:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-53) [4960f98a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 57630c66 >2013-03-08 11:10:00,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [4960f98a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 19a95701 >2013-03-08 11:10:00,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [4960f98a] FINISH, GetHardwareInfoVDSCommand, log id: 19a95701 >2013-03-08 11:10:00,458 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-53) [5bfc3c95] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:10:00,460 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-53) [670351bb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:10:00,462 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-53) [670351bb] FINISH, ActivateVdsVDSCommand, log id: 57630c66 >2013-03-08 11:10:00,481 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-53) [670351bb] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:10:00,482 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-53) [670351bb] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 11:10:00,483 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [670351bb] Autorecovering 1 storage domains >2013-03-08 11:10:00,483 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [670351bb] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 11:10:00,484 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-53) [294d68be] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:10:00,484 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-53) [294d68be] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 11:10 AM >2013-03-08 11:10:00,487 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-53) [294d68be] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 11:10 AM >2013-03-08 11:10:02,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [2fe8822] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4500ca0b >2013-03-08 11:10:02,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [2fe8822] FINISH, GetHardwareInfoVDSCommand, log id: 4500ca0b >2013-03-08 11:10:02,670 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-81) [40060b64] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 11:10:02,674 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-81) [21a2ac77] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 11:10:02,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2c71e793 >2013-03-08 11:11:05,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 11:11:05,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:11:05,671 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 11:11:05,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 2c71e793 >2013-03-08 11:11:05,675 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [51acc8b2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:11:05,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [51acc8b2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 55ae3f1a >2013-03-08 11:11:05,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [51acc8b2] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 11:11:05,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [51acc8b2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 55ae3f1a >2013-03-08 11:11:05,691 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [51acc8b2] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 11:11:05,692 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [21a2ac77] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 11:11:05,701 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-81) [3616a94f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:11:05,703 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [3616a94f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4bcc975b >2013-03-08 11:11:05,704 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [3616a94f] FINISH, SetVdsStatusVDSCommand, log id: 4bcc975b >2013-03-08 11:11:05,710 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-81) [3616a94f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 11:11:05,731 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-81) [1b05b75c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:11:05,733 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-81) [2ff6ef51] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:11:05,735 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-81) [2ff6ef51] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 11:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [6c01ec2e] Autorecovering 1 hosts >2013-03-08 11:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [6c01ec2e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [3edf4609] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [3edf4609] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [3edf4609] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7ab53c41 >2013-03-08 11:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [3edf4609] FINISH, SetVdsStatusVDSCommand, log id: 7ab53c41 >2013-03-08 11:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [3edf4609] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7c4245d1 >2013-03-08 11:15:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [3edf4609] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3df2b43 >2013-03-08 11:15:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [3edf4609] FINISH, GetHardwareInfoVDSCommand, log id: 3df2b43 >2013-03-08 11:15:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-63) [569ddbbc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:15:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-63) [476ad462] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:15:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [476ad462] FINISH, ActivateVdsVDSCommand, log id: 7c4245d1 >2013-03-08 11:15:00,111 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [476ad462] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:15:00,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-63) [476ad462] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 11:15:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [476ad462] Autorecovering 1 storage domains >2013-03-08 11:15:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [476ad462] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 11:15:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [7a1f987e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:15:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [7a1f987e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 11:15 AM >2013-03-08 11:15:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [7a1f987e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 11:15 AM >2013-03-08 11:15:02,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [550cdc8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1b5b5f86 >2013-03-08 11:15:02,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [550cdc8] FINISH, GetHardwareInfoVDSCommand, log id: 1b5b5f86 >2013-03-08 11:15:02,984 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-58) [6b2d5383] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 11:15:02,988 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-58) [276291f6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 11:15:02,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1b38b948 >2013-03-08 11:16:06,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 11:16:06,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:16:06,987 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 11:16:06,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 1b38b948 >2013-03-08 11:16:06,991 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [640f7b00] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:16:07,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [640f7b00] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 619e5463 >2013-03-08 11:16:07,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [640f7b00] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 11:16:07,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [640f7b00] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 619e5463 >2013-03-08 11:16:07,007 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [640f7b00] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 11:16:07,007 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [276291f6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 11:16:07,017 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-58) [47070e5c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:16:07,018 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [47070e5c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 58416a2c >2013-03-08 11:16:07,020 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [47070e5c] FINISH, SetVdsStatusVDSCommand, log id: 58416a2c >2013-03-08 11:16:07,026 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-58) [47070e5c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 11:16:07,047 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-58) [5bb695fa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:16:07,050 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-58) [ee9c6e6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:16:07,051 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-58) [ee9c6e6] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 11:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [633810d2] Autorecovering 1 hosts >2013-03-08 11:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [633810d2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [720086c9] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [720086c9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [720086c9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 73d4c950 >2013-03-08 11:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [720086c9] FINISH, SetVdsStatusVDSCommand, log id: 73d4c950 >2013-03-08 11:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-24) [720086c9] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 39af1b60 >2013-03-08 11:20:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [720086c9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 54d796c5 >2013-03-08 11:20:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [720086c9] FINISH, GetHardwareInfoVDSCommand, log id: 54d796c5 >2013-03-08 11:20:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-24) [57f70572] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:20:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-24) [58905513] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:20:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-24) [58905513] FINISH, ActivateVdsVDSCommand, log id: 39af1b60 >2013-03-08 11:20:00,221 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [58905513] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:20:00,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-24) [58905513] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 11:20:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [58905513] Autorecovering 1 storage domains >2013-03-08 11:20:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [58905513] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 11:20:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [4ed5b2f0] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:20:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [4ed5b2f0] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 11:20 AM >2013-03-08 11:20:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [4ed5b2f0] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 11:20 AM >2013-03-08 11:20:01,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [73c289b8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 329c9126 >2013-03-08 11:20:01,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [73c289b8] FINISH, GetHardwareInfoVDSCommand, log id: 329c9126 >2013-03-08 11:20:01,289 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-33) [58208b05] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 11:20:01,292 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-33) [6ef21c1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 11:20:01,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2d3901d8 >2013-03-08 11:21:04,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 11:21:04,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:21:04,295 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 11:21:04,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2d3901d8 >2013-03-08 11:21:04,299 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [3bd7fb4e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:21:04,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [3bd7fb4e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 371f1d80 >2013-03-08 11:21:04,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [3bd7fb4e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 11:21:04,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [3bd7fb4e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 371f1d80 >2013-03-08 11:21:04,318 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [3bd7fb4e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 11:21:04,319 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [6ef21c1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 11:21:04,328 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-33) [35aa5794] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:21:04,329 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [35aa5794] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 409de07 >2013-03-08 11:21:04,330 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [35aa5794] FINISH, SetVdsStatusVDSCommand, log id: 409de07 >2013-03-08 11:21:04,336 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-33) [35aa5794] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 11:21:04,357 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-33) [44ad80ed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:21:04,359 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-33) [3bfb99c0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:21:04,361 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-33) [3bfb99c0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 11:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [78204549] Autorecovering 1 hosts >2013-03-08 11:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [78204549] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [54733c0e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [54733c0e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:25:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [54733c0e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 547acce6 >2013-03-08 11:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [54733c0e] FINISH, SetVdsStatusVDSCommand, log id: 547acce6 >2013-03-08 11:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-90) [54733c0e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4301f632 >2013-03-08 11:25:00,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [54733c0e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 10d09ec2 >2013-03-08 11:25:00,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [54733c0e] FINISH, GetHardwareInfoVDSCommand, log id: 10d09ec2 >2013-03-08 11:25:00,190 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-90) [3befd139] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:25:00,193 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-90) [3adae54a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:25:00,194 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-90) [3adae54a] FINISH, ActivateVdsVDSCommand, log id: 4301f632 >2013-03-08 11:25:00,213 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [3adae54a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:25:00,214 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [3adae54a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 11:25:00,215 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [3adae54a] Autorecovering 1 storage domains >2013-03-08 11:25:00,215 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [3adae54a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 11:25:00,216 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [13c58c68] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:25:00,216 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [13c58c68] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 11:25 AM >2013-03-08 11:25:00,219 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [13c58c68] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 11:25 AM >2013-03-08 11:25:01,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [1be500d9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: b22a180 >2013-03-08 11:25:01,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [1be500d9] FINISH, GetHardwareInfoVDSCommand, log id: b22a180 >2013-03-08 11:25:01,607 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-76) [32c353ff] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 11:25:01,610 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-76) [ab59bd3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 11:25:01,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2b2f705d >2013-03-08 11:26:06,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 11:26:06,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:26:06,647 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 11:26:06,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2b2f705d >2013-03-08 11:26:06,650 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [158885a0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:26:06,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [158885a0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 260a3d91 >2013-03-08 11:26:06,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [158885a0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 11:26:06,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [158885a0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 260a3d91 >2013-03-08 11:26:06,666 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [158885a0] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 11:26:06,667 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [ab59bd3] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 11:26:06,686 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-76) [14549345] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:26:06,688 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [14549345] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 659891fb >2013-03-08 11:26:06,689 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [14549345] FINISH, SetVdsStatusVDSCommand, log id: 659891fb >2013-03-08 11:26:06,695 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-76) [14549345] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 11:26:06,716 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-76) [92050b1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:26:06,719 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-76) [6edef405] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:26:06,720 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-76) [6edef405] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 11:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [2ce40ea1] Autorecovering 1 hosts >2013-03-08 11:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [2ce40ea1] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-98) [73d37ab0] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-98) [73d37ab0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-98) [73d37ab0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 21b4880e >2013-03-08 11:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-98) [73d37ab0] FINISH, SetVdsStatusVDSCommand, log id: 21b4880e >2013-03-08 11:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-98) [73d37ab0] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 515d46c9 >2013-03-08 11:30:00,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-98) [73d37ab0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3f1e70e6 >2013-03-08 11:30:00,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-98) [73d37ab0] FINISH, GetHardwareInfoVDSCommand, log id: 3f1e70e6 >2013-03-08 11:30:00,207 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-98) [5bc37f89] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:30:00,210 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-98) [1fef417c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:30:00,211 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-98) [1fef417c] FINISH, ActivateVdsVDSCommand, log id: 515d46c9 >2013-03-08 11:30:00,230 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-98) [1fef417c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:30:00,231 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-98) [1fef417c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 11:30:00,232 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [1fef417c] Autorecovering 1 storage domains >2013-03-08 11:30:00,232 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [1fef417c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 11:30:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-98) [37e37665] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:30:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-98) [37e37665] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 11:30 AM >2013-03-08 11:30:00,236 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-98) [37e37665] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 11:30 AM >2013-03-08 11:30:00,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [4a6737d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 59efdca8 >2013-03-08 11:30:00,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [4a6737d] FINISH, GetHardwareInfoVDSCommand, log id: 59efdca8 >2013-03-08 11:30:00,964 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-72) [2c2faf2f] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 11:30:00,968 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-72) [203e13d5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 11:30:00,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4b4e7263 >2013-03-08 11:30:33,738 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-77) [1e8ba581] Start refreshing all users data >2013-03-08 11:30:34,680 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-92) [2f044ed4] Attempting to update VMs/Templates Ovf. >2013-03-08 11:30:34,681 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-92) [2f044ed4] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 11:30:34,681 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-92) [2f044ed4] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 11:30:34,682 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-92) [2f044ed4] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 11:30:34,682 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-92) [2f044ed4] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 11:30:34,682 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-92) [2f044ed4] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 11:30:34,683 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-92) [2f044ed4] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 11:31:05,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 11:31:05,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:31:05,990 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 11:31:05,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 4b4e7263 >2013-03-08 11:31:05,993 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [51739940] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:31:06,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [51739940] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3919cd4a >2013-03-08 11:31:06,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [51739940] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 11:31:06,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [51739940] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3919cd4a >2013-03-08 11:31:06,018 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [51739940] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 11:31:06,018 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [203e13d5] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 11:31:06,028 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-72) [41c3e2f6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:31:06,029 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [41c3e2f6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 47692745 >2013-03-08 11:31:06,030 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [41c3e2f6] FINISH, SetVdsStatusVDSCommand, log id: 47692745 >2013-03-08 11:31:06,037 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [41c3e2f6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 11:31:06,056 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [cbb6909] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:31:06,059 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [497e931f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:31:06,060 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-72) [497e931f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 11:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [19d932c5] Autorecovering 1 hosts >2013-03-08 11:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [19d932c5] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [2b8a36b1] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [2b8a36b1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [2b8a36b1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 488f752e >2013-03-08 11:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [2b8a36b1] FINISH, SetVdsStatusVDSCommand, log id: 488f752e >2013-03-08 11:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [2b8a36b1] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 783bce2f >2013-03-08 11:35:00,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [2b8a36b1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2dc5511d >2013-03-08 11:35:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [2b8a36b1] FINISH, GetHardwareInfoVDSCommand, log id: 2dc5511d >2013-03-08 11:35:00,184 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-45) [75bed292] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:35:00,186 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-45) [3a726d2d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:35:00,187 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [3a726d2d] FINISH, ActivateVdsVDSCommand, log id: 783bce2f >2013-03-08 11:35:00,207 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [3a726d2d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:35:00,208 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-45) [3a726d2d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 11:35:00,211 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [3a726d2d] Autorecovering 1 storage domains >2013-03-08 11:35:00,211 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [3a726d2d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 11:35:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [e271f32] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:35:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [e271f32] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 11:35 AM >2013-03-08 11:35:00,217 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [e271f32] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 11:35 AM >2013-03-08 11:35:00,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-62) [25797610] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 202b465b >2013-03-08 11:35:00,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-62) [25797610] FINISH, GetHardwareInfoVDSCommand, log id: 202b465b >2013-03-08 11:35:00,306 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-62) [22a1c00a] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 11:35:00,309 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-62) [11096161] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 11:35:00,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 53685b88 >2013-03-08 11:36:04,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 11:36:04,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:36:04,613 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 11:36:04,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 53685b88 >2013-03-08 11:36:04,617 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [23f34094] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:36:04,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [23f34094] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 700e60fd >2013-03-08 11:36:04,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [23f34094] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 11:36:04,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [23f34094] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 700e60fd >2013-03-08 11:36:04,639 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [23f34094] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 11:36:04,639 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [11096161] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 11:36:04,649 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-62) [47c76f5b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:36:04,650 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-62) [47c76f5b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6e0df80c >2013-03-08 11:36:04,651 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-62) [47c76f5b] FINISH, SetVdsStatusVDSCommand, log id: 6e0df80c >2013-03-08 11:36:04,658 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-62) [47c76f5b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 11:36:04,684 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-62) [55a8ba0d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:36:04,686 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-62) [128d0b36] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:36:04,687 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-62) [128d0b36] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 11:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [3a756cd6] Autorecovering 1 hosts >2013-03-08 11:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [3a756cd6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [41dcc17a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [41dcc17a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:40:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [41dcc17a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 36f3eeeb >2013-03-08 11:40:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [41dcc17a] FINISH, SetVdsStatusVDSCommand, log id: 36f3eeeb >2013-03-08 11:40:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-20) [41dcc17a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 164689a4 >2013-03-08 11:40:00,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [41dcc17a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5bba08c9 >2013-03-08 11:40:00,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [41dcc17a] FINISH, GetHardwareInfoVDSCommand, log id: 5bba08c9 >2013-03-08 11:40:00,453 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-20) [1f1b5010] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:40:00,455 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-20) [5c10cf85] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:40:00,457 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-20) [5c10cf85] FINISH, ActivateVdsVDSCommand, log id: 164689a4 >2013-03-08 11:40:00,477 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [5c10cf85] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:40:00,477 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-20) [5c10cf85] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 11:40:00,479 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [5c10cf85] Autorecovering 1 storage domains >2013-03-08 11:40:00,479 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [5c10cf85] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 11:40:00,479 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [76c1c41c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:40:00,480 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [76c1c41c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 11:40 AM >2013-03-08 11:40:00,482 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [76c1c41c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 11:40 AM >2013-03-08 11:40:01,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-29) [75e7377b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: f4eba83 >2013-03-08 11:40:01,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-29) [75e7377b] FINISH, GetHardwareInfoVDSCommand, log id: f4eba83 >2013-03-08 11:40:01,935 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-29) [34f4b482] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 11:40:01,939 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-29) [2a583bd5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 11:40:01,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 766377d3 >2013-03-08 11:41:07,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 11:41:07,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:41:07,921 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 11:41:07,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 766377d3 >2013-03-08 11:41:07,924 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [b378994] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:41:07,938 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [b378994] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 622d5fb8 >2013-03-08 11:41:07,938 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [b378994] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 11:41:07,938 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [b378994] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 622d5fb8 >2013-03-08 11:41:07,940 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [b378994] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 11:41:07,942 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [2a583bd5] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 11:41:07,952 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-29) [7c7f3b33] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:41:07,954 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-29) [7c7f3b33] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 34eed03a >2013-03-08 11:41:07,955 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-29) [7c7f3b33] FINISH, SetVdsStatusVDSCommand, log id: 34eed03a >2013-03-08 11:41:07,961 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-29) [7c7f3b33] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 11:41:07,982 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-29) [1252441a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:41:07,985 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-29) [21f8e886] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:41:07,986 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-29) [21f8e886] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 11:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [11989d9a] Autorecovering 1 hosts >2013-03-08 11:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [11989d9a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [52278845] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [52278845] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [52278845] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 657dd660 >2013-03-08 11:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [52278845] FINISH, SetVdsStatusVDSCommand, log id: 657dd660 >2013-03-08 11:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [52278845] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2f97aa34 >2013-03-08 11:45:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [52278845] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2e78720b >2013-03-08 11:45:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [52278845] FINISH, GetHardwareInfoVDSCommand, log id: 2e78720b >2013-03-08 11:45:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [64a974a9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:45:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [2eb6bfcd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:45:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [2eb6bfcd] FINISH, ActivateVdsVDSCommand, log id: 2f97aa34 >2013-03-08 11:45:00,111 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [2eb6bfcd] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:45:00,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [2eb6bfcd] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 11:45:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [2eb6bfcd] Autorecovering 1 storage domains >2013-03-08 11:45:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [2eb6bfcd] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 11:45:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [353e723a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:45:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [353e723a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 11:45 AM >2013-03-08 11:45:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [353e723a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 11:45 AM >2013-03-08 11:45:02,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-6) [7fa35030] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 9360b7f >2013-03-08 11:45:02,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-6) [7fa35030] FINISH, GetHardwareInfoVDSCommand, log id: 9360b7f >2013-03-08 11:45:02,231 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-6) [5a16d44] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 11:45:02,234 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-6) [6e5f0e01] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 11:45:02,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 727868e9 >2013-03-08 11:46:07,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 11:46:07,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:46:07,211 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 11:46:07,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 727868e9 >2013-03-08 11:46:07,215 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [622484d2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:46:07,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [622484d2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 71e249d8 >2013-03-08 11:46:07,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [622484d2] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 11:46:07,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [622484d2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 71e249d8 >2013-03-08 11:46:07,242 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [6e5f0e01] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 11:46:07,253 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-6) [22bb8fc5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:46:07,255 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-6) [22bb8fc5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 41d0d8c2 >2013-03-08 11:46:07,256 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-6) [22bb8fc5] FINISH, SetVdsStatusVDSCommand, log id: 41d0d8c2 >2013-03-08 11:46:07,264 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-6) [22bb8fc5] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 11:46:07,285 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-6) [6871f4ed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:46:07,287 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-6) [5355f28c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:46:07,289 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-6) [5355f28c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 11:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [23d73427] Autorecovering 1 hosts >2013-03-08 11:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [23d73427] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-48) [11881463] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-48) [11881463] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [11881463] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 20408d52 >2013-03-08 11:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [11881463] FINISH, SetVdsStatusVDSCommand, log id: 20408d52 >2013-03-08 11:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-48) [11881463] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2034f07 >2013-03-08 11:50:00,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [11881463] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 313852da >2013-03-08 11:50:00,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [11881463] FINISH, GetHardwareInfoVDSCommand, log id: 313852da >2013-03-08 11:50:00,188 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-48) [48898799] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:50:00,190 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-48) [7137c2ff] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:50:00,192 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-48) [7137c2ff] FINISH, ActivateVdsVDSCommand, log id: 2034f07 >2013-03-08 11:50:00,211 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-48) [7137c2ff] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:50:00,212 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-48) [7137c2ff] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 11:50:00,213 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [7137c2ff] Autorecovering 1 storage domains >2013-03-08 11:50:00,213 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [7137c2ff] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 11:50:00,214 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-48) [5e26a1e0] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:50:00,214 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-48) [5e26a1e0] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 11:50 AM >2013-03-08 11:50:00,217 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-48) [5e26a1e0] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 11:50 AM >2013-03-08 11:50:01,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [74dcecd3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 38a11ae5 >2013-03-08 11:50:01,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [74dcecd3] FINISH, GetHardwareInfoVDSCommand, log id: 38a11ae5 >2013-03-08 11:50:01,535 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-100) [6e521f67] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 11:50:01,539 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-100) [40d6f693] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 11:50:01,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5422f05e >2013-03-08 11:51:04,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 11:51:04,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:51:04,534 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 11:51:04,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 5422f05e >2013-03-08 11:51:04,538 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [696737cd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:51:04,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [696737cd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6659a5e5 >2013-03-08 11:51:04,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [696737cd] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 11:51:04,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [696737cd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6659a5e5 >2013-03-08 11:51:04,556 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [696737cd] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 11:51:04,557 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [40d6f693] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 11:51:04,567 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-100) [23193a20] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:51:04,569 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [23193a20] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2415124e >2013-03-08 11:51:04,570 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [23193a20] FINISH, SetVdsStatusVDSCommand, log id: 2415124e >2013-03-08 11:51:04,576 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-100) [23193a20] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 11:51:04,598 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-100) [14df996f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:51:04,600 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-100) [49ebe838] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:51:04,601 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-100) [49ebe838] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 11:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [375b599c] Autorecovering 1 hosts >2013-03-08 11:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [375b599c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [73945797] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [73945797] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [73945797] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5c893799 >2013-03-08 11:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [73945797] FINISH, SetVdsStatusVDSCommand, log id: 5c893799 >2013-03-08 11:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [73945797] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 66897942 >2013-03-08 11:55:00,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [73945797] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7eba8f78 >2013-03-08 11:55:00,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [73945797] FINISH, GetHardwareInfoVDSCommand, log id: 7eba8f78 >2013-03-08 11:55:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-69) [188ff53] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:55:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-69) [b20d0bd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:55:00,206 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [b20d0bd] FINISH, ActivateVdsVDSCommand, log id: 66897942 >2013-03-08 11:55:00,225 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [b20d0bd] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 11:55:00,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-69) [b20d0bd] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 11:55:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [b20d0bd] Autorecovering 1 storage domains >2013-03-08 11:55:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [b20d0bd] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 11:55:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [652c6408] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:55:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [652c6408] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 11:55 AM >2013-03-08 11:55:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [652c6408] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 11:55 AM >2013-03-08 11:55:01,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [c72cc82] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 36d091b >2013-03-08 11:55:01,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [c72cc82] FINISH, GetHardwareInfoVDSCommand, log id: 36d091b >2013-03-08 11:55:01,848 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-10) [2e840c24] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 11:55:01,851 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-10) [1b9823f6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 11:55:01,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 475c0ac9 >2013-03-08 11:56:08,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 11:56:08,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 11:56:08,900 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 11:56:08,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 475c0ac9 >2013-03-08 11:56:08,904 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [379e733c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 11:56:08,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [379e733c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7a458246 >2013-03-08 11:56:08,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [379e733c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 11:56:08,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [379e733c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7a458246 >2013-03-08 11:56:08,920 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [379e733c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 11:56:08,920 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [1b9823f6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 11:56:08,931 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-10) [3d1695bb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:56:08,932 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [3d1695bb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: a2bbb69 >2013-03-08 11:56:08,933 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [3d1695bb] FINISH, SetVdsStatusVDSCommand, log id: a2bbb69 >2013-03-08 11:56:08,940 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-10) [3d1695bb] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 11:56:08,961 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-10) [a0718a3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:56:08,964 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-10) [3719a4cc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 11:56:08,965 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-10) [3719a4cc] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 12:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [f047e82] Autorecovering 1 hosts >2013-03-08 12:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [f047e82] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [7e94c8ae] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [7e94c8ae] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [7e94c8ae] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3761e820 >2013-03-08 12:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [7e94c8ae] FINISH, SetVdsStatusVDSCommand, log id: 3761e820 >2013-03-08 12:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-93) [7e94c8ae] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 43fc4b19 >2013-03-08 12:00:00,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [7e94c8ae] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 20b6fadc >2013-03-08 12:00:00,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [7e94c8ae] FINISH, GetHardwareInfoVDSCommand, log id: 20b6fadc >2013-03-08 12:00:00,185 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [74f3bd9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:00:00,188 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [48796990] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:00:00,189 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-93) [48796990] FINISH, ActivateVdsVDSCommand, log id: 43fc4b19 >2013-03-08 12:00:00,209 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [48796990] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:00:00,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-93) [48796990] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 12:00:00,211 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [48796990] Autorecovering 1 storage domains >2013-03-08 12:00:00,211 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [48796990] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 12:00:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [5bb0d0b6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:00:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [5bb0d0b6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:00 PM >2013-03-08 12:00:00,215 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [5bb0d0b6] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:00 PM >2013-03-08 12:00:00,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [2beb7743] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6cc8cb09 >2013-03-08 12:00:00,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [2beb7743] FINISH, GetHardwareInfoVDSCommand, log id: 6cc8cb09 >2013-03-08 12:00:00,292 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-43) [2ee7daa8] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 12:00:00,299 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-43) [4af0712d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 12:00:00,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 569abdfe >2013-03-08 12:01:05,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 12:01:05,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:01:05,285 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 12:01:05,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 569abdfe >2013-03-08 12:01:05,290 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [1bb0438e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:01:05,304 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [1bb0438e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 672b04cf >2013-03-08 12:01:05,304 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [1bb0438e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 12:01:05,304 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [1bb0438e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 672b04cf >2013-03-08 12:01:05,306 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [1bb0438e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 12:01:05,307 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [4af0712d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 12:01:05,316 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-43) [185fba61] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:01:05,318 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [185fba61] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 61cbb7a6 >2013-03-08 12:01:05,319 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [185fba61] FINISH, SetVdsStatusVDSCommand, log id: 61cbb7a6 >2013-03-08 12:01:05,324 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [185fba61] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 12:01:05,350 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [329b5983] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:01:05,352 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [7b066bd7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:01:05,354 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-43) [7b066bd7] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 12:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [2f044ed4] Autorecovering 1 hosts >2013-03-08 12:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [2f044ed4] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [a5e3d27] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [a5e3d27] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [a5e3d27] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7ce8fdcb >2013-03-08 12:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [a5e3d27] FINISH, SetVdsStatusVDSCommand, log id: 7ce8fdcb >2013-03-08 12:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-92) [a5e3d27] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 51de383d >2013-03-08 12:05:00,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [a5e3d27] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2db58398 >2013-03-08 12:05:00,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [a5e3d27] FINISH, GetHardwareInfoVDSCommand, log id: 2db58398 >2013-03-08 12:05:00,236 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-92) [26df7317] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:05:00,238 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-92) [cb52547] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:05:00,239 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-92) [cb52547] FINISH, ActivateVdsVDSCommand, log id: 51de383d >2013-03-08 12:05:00,262 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [cb52547] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:05:00,263 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-92) [cb52547] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 12:05:00,264 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [cb52547] Autorecovering 1 storage domains >2013-03-08 12:05:00,264 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [cb52547] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 12:05:00,265 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [25f71f5a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:05:00,265 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [25f71f5a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:05 PM >2013-03-08 12:05:00,268 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [25f71f5a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:05 PM >2013-03-08 12:05:02,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [6f33e1c0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2427d857 >2013-03-08 12:05:02,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [6f33e1c0] FINISH, GetHardwareInfoVDSCommand, log id: 2427d857 >2013-03-08 12:05:02,598 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-64) [76964676] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 12:05:02,602 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-64) [39624b1e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 12:05:02,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 45924608 >2013-03-08 12:06:07,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 12:06:07,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:06:07,601 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 12:06:07,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 45924608 >2013-03-08 12:06:07,606 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [61d6f1d9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:06:07,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [61d6f1d9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 50196160 >2013-03-08 12:06:07,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [61d6f1d9] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 12:06:07,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [61d6f1d9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 50196160 >2013-03-08 12:06:07,623 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [61d6f1d9] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 12:06:07,623 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [39624b1e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 12:06:07,633 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-64) [6f4355d5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:06:07,634 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [6f4355d5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 337cf054 >2013-03-08 12:06:07,635 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [6f4355d5] FINISH, SetVdsStatusVDSCommand, log id: 337cf054 >2013-03-08 12:06:07,642 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-64) [6f4355d5] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 12:06:07,662 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-64) [5e808198] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:06:07,664 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-64) [12425f9d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:06:07,666 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-64) [12425f9d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 12:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [24d08e02] Autorecovering 1 hosts >2013-03-08 12:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [24d08e02] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [7dea83d6] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [7dea83d6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:10:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [7dea83d6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5d99a9ee >2013-03-08 12:10:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [7dea83d6] FINISH, SetVdsStatusVDSCommand, log id: 5d99a9ee >2013-03-08 12:10:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-68) [7dea83d6] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 84ac27b >2013-03-08 12:10:00,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [7dea83d6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2dcb6da5 >2013-03-08 12:10:00,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [7dea83d6] FINISH, GetHardwareInfoVDSCommand, log id: 2dcb6da5 >2013-03-08 12:10:00,630 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-68) [3443dc0f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:10:00,633 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-68) [53cb9199] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:10:00,634 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-68) [53cb9199] FINISH, ActivateVdsVDSCommand, log id: 84ac27b >2013-03-08 12:10:00,661 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [53cb9199] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:10:00,662 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-68) [53cb9199] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 12:10:00,663 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [53cb9199] Autorecovering 1 storage domains >2013-03-08 12:10:00,663 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [53cb9199] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 12:10:00,664 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [eeec576] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:10:00,664 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [eeec576] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:10 PM >2013-03-08 12:10:00,667 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [eeec576] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:10 PM >2013-03-08 12:10:01,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [787a75db] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3bd7ec99 >2013-03-08 12:10:01,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [787a75db] FINISH, GetHardwareInfoVDSCommand, log id: 3bd7ec99 >2013-03-08 12:10:01,915 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-75) [72d9d49a] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 12:10:01,924 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-75) [4f848e58] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 12:10:01,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 814bc42 >2013-03-08 12:11:07,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 12:11:07,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:11:07,927 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 12:11:07,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 814bc42 >2013-03-08 12:11:07,931 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [92b2b79] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:11:07,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [92b2b79] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 65838918 >2013-03-08 12:11:07,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [92b2b79] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 12:11:07,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [92b2b79] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 65838918 >2013-03-08 12:11:07,948 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [92b2b79] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 12:11:07,948 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [4f848e58] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 12:11:07,957 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-75) [2fecc7eb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:11:07,959 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [2fecc7eb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 81df831 >2013-03-08 12:11:07,960 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [2fecc7eb] FINISH, SetVdsStatusVDSCommand, log id: 81df831 >2013-03-08 12:11:07,966 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-75) [2fecc7eb] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 12:11:07,987 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [29055365] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:11:07,989 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [520061e8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:11:07,991 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-75) [520061e8] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 12:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [19f642c4] Autorecovering 1 hosts >2013-03-08 12:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [19f642c4] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [abce0cb] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [abce0cb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [abce0cb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1ef01bd >2013-03-08 12:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [abce0cb] FINISH, SetVdsStatusVDSCommand, log id: 1ef01bd >2013-03-08 12:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [abce0cb] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3f084af2 >2013-03-08 12:15:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [abce0cb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 68bd973f >2013-03-08 12:15:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [abce0cb] FINISH, GetHardwareInfoVDSCommand, log id: 68bd973f >2013-03-08 12:15:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-14) [1702a739] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:15:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-14) [537ee786] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:15:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [537ee786] FINISH, ActivateVdsVDSCommand, log id: 3f084af2 >2013-03-08 12:15:00,114 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [537ee786] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:15:00,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-14) [537ee786] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 12:15:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [537ee786] Autorecovering 1 storage domains >2013-03-08 12:15:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [537ee786] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 12:15:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [2cc208aa] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:15:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [2cc208aa] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:15 PM >2013-03-08 12:15:00,125 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [2cc208aa] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:15 PM >2013-03-08 12:15:02,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [71cd7733] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6918f33d >2013-03-08 12:15:02,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [71cd7733] FINISH, GetHardwareInfoVDSCommand, log id: 6918f33d >2013-03-08 12:15:02,233 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-61) [28f78979] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 12:15:02,236 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-61) [43306183] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 12:15:02,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 403f3f36 >2013-03-08 12:16:06,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 12:16:06,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:16:06,228 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 12:16:06,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 403f3f36 >2013-03-08 12:16:06,232 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [46a5b915] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:16:06,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [46a5b915] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 64f8fc8f >2013-03-08 12:16:06,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [46a5b915] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 12:16:06,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [46a5b915] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 64f8fc8f >2013-03-08 12:16:06,249 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [46a5b915] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 12:16:06,249 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [43306183] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 12:16:06,259 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-61) [604ffb3b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:16:06,261 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [604ffb3b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 49b31e76 >2013-03-08 12:16:06,262 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [604ffb3b] FINISH, SetVdsStatusVDSCommand, log id: 49b31e76 >2013-03-08 12:16:06,268 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-61) [604ffb3b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 12:16:06,288 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-61) [78ad2429] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:16:06,290 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-61) [1a8af0b6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:16:06,292 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-61) [1a8af0b6] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 12:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [4390d057] Autorecovering 1 hosts >2013-03-08 12:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [4390d057] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [720964ef] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [720964ef] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [720964ef] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6cf8121a >2013-03-08 12:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [720964ef] FINISH, SetVdsStatusVDSCommand, log id: 6cf8121a >2013-03-08 12:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-70) [720964ef] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: b3b290c >2013-03-08 12:20:00,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [720964ef] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7d6952b3 >2013-03-08 12:20:00,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [720964ef] FINISH, GetHardwareInfoVDSCommand, log id: 7d6952b3 >2013-03-08 12:20:00,208 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-70) [78179d8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:20:00,211 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-70) [408014a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:20:00,212 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-70) [408014a] FINISH, ActivateVdsVDSCommand, log id: b3b290c >2013-03-08 12:20:00,237 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [408014a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:20:00,238 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-70) [408014a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 12:20:00,239 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [408014a] Autorecovering 1 storage domains >2013-03-08 12:20:00,239 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [408014a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 12:20:00,241 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [f64b7f4] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:20:00,241 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [f64b7f4] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:20 PM >2013-03-08 12:20:00,246 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [f64b7f4] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:20 PM >2013-03-08 12:20:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [448102d1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5e3aef13 >2013-03-08 12:20:00,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [448102d1] FINISH, GetHardwareInfoVDSCommand, log id: 5e3aef13 >2013-03-08 12:20:00,534 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-88) [9be01f] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 12:20:00,541 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-88) [17b4cb1a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 12:20:00,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 376895f0 >2013-03-08 12:21:04,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 12:21:04,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:21:04,532 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 12:21:04,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 376895f0 >2013-03-08 12:21:04,536 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [391f4f1a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:21:04,553 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [391f4f1a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 14bce050 >2013-03-08 12:21:04,553 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [391f4f1a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 12:21:04,553 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [391f4f1a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 14bce050 >2013-03-08 12:21:04,555 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [391f4f1a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 12:21:04,556 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [17b4cb1a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 12:21:04,566 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-88) [6f9f830d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:21:04,572 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [6f9f830d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4105a8f1 >2013-03-08 12:21:04,573 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [6f9f830d] FINISH, SetVdsStatusVDSCommand, log id: 4105a8f1 >2013-03-08 12:21:04,580 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-88) [6f9f830d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 12:21:04,601 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-88) [8b25440] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:21:04,603 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-88) [756a6735] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:21:04,605 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-88) [756a6735] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 12:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [2ef4d8bf] Autorecovering 1 hosts >2013-03-08 12:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [2ef4d8bf] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [6f946723] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [6f946723] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [6f946723] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7ebd1547 >2013-03-08 12:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [6f946723] FINISH, SetVdsStatusVDSCommand, log id: 7ebd1547 >2013-03-08 12:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [6f946723] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 28005264 >2013-03-08 12:25:00,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [6f946723] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 77f50aae >2013-03-08 12:25:00,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [6f946723] FINISH, GetHardwareInfoVDSCommand, log id: 77f50aae >2013-03-08 12:25:00,235 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-79) [5de0b2b0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:25:00,237 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-79) [285f7ec1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:25:00,239 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [285f7ec1] FINISH, ActivateVdsVDSCommand, log id: 28005264 >2013-03-08 12:25:00,258 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [285f7ec1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:25:00,262 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [285f7ec1] Autorecovering 1 storage domains >2013-03-08 12:25:00,264 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [285f7ec1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 12:25:00,269 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [b60a8e3] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:25:00,270 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [b60a8e3] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:25 PM >2013-03-08 12:25:00,273 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [b60a8e3] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:25 PM >2013-03-08 12:25:01,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [7827ffd1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5e273294 >2013-03-08 12:25:01,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [7827ffd1] FINISH, GetHardwareInfoVDSCommand, log id: 5e273294 >2013-03-08 12:25:01,857 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-17) [34f2c487] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 12:25:01,861 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-17) [5c0d43af] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 12:25:01,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 29c34892 >2013-03-08 12:26:06,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 12:26:06,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:26:06,913 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 12:26:06,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 29c34892 >2013-03-08 12:26:06,917 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [1b2df93f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:26:06,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [1b2df93f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2f1e9c38 >2013-03-08 12:26:06,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [1b2df93f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 12:26:06,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [1b2df93f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2f1e9c38 >2013-03-08 12:26:06,933 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [1b2df93f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 12:26:06,934 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [5c0d43af] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 12:26:06,943 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-17) [7edaecac] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:26:06,945 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [7edaecac] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 51208e77 >2013-03-08 12:26:06,946 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [7edaecac] FINISH, SetVdsStatusVDSCommand, log id: 51208e77 >2013-03-08 12:26:06,952 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [7edaecac] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 12:26:06,991 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [3bf130bb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:26:06,993 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [7dc0a8a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:26:06,994 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-17) [7dc0a8a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 12:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [6ffac8dc] Autorecovering 1 hosts >2013-03-08 12:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [6ffac8dc] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-67) [5cc9bb38] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-67) [5cc9bb38] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [5cc9bb38] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2f91b2d9 >2013-03-08 12:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [5cc9bb38] FINISH, SetVdsStatusVDSCommand, log id: 2f91b2d9 >2013-03-08 12:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-67) [5cc9bb38] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1accc0b0 >2013-03-08 12:30:00,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [5cc9bb38] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2dde22c9 >2013-03-08 12:30:00,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [5cc9bb38] FINISH, GetHardwareInfoVDSCommand, log id: 2dde22c9 >2013-03-08 12:30:00,191 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-67) [2afe87ed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:30:00,194 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-67) [125ac718] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:30:00,195 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-67) [125ac718] FINISH, ActivateVdsVDSCommand, log id: 1accc0b0 >2013-03-08 12:30:00,214 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-67) [125ac718] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:30:00,215 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-67) [125ac718] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 12:30:00,216 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [125ac718] Autorecovering 1 storage domains >2013-03-08 12:30:00,216 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [125ac718] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 12:30:00,217 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-67) [f349d04] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:30:00,217 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-67) [f349d04] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:30 PM >2013-03-08 12:30:00,220 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-67) [f349d04] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:30 PM >2013-03-08 12:30:01,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [515db1bc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2dd54d65 >2013-03-08 12:30:01,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [515db1bc] FINISH, GetHardwareInfoVDSCommand, log id: 2dd54d65 >2013-03-08 12:30:01,241 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-87) [3c8c2009] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 12:30:01,244 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-87) [45b2955f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 12:30:01,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5db0650b >2013-03-08 12:30:33,740 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-42) [1d287d1c] Start refreshing all users data >2013-03-08 12:30:34,684 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-40) [90e1ee3] Attempting to update VMs/Templates Ovf. >2013-03-08 12:30:34,685 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-40) [90e1ee3] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 12:30:34,686 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-40) [90e1ee3] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 12:30:34,686 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-40) [90e1ee3] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 12:30:34,686 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-40) [90e1ee3] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 12:30:34,686 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-40) [90e1ee3] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 12:30:34,687 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-40) [90e1ee3] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 12:31:06,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 12:31:06,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:31:06,324 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 12:31:06,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 5db0650b >2013-03-08 12:31:06,327 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [2759c4a2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:31:06,342 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [2759c4a2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 705128eb >2013-03-08 12:31:06,342 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [2759c4a2] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 12:31:06,342 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [2759c4a2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 705128eb >2013-03-08 12:31:06,344 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [2759c4a2] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 12:31:06,344 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [45b2955f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 12:31:06,354 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-87) [16fa6415] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:31:06,356 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [16fa6415] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5db39b17 >2013-03-08 12:31:06,357 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [16fa6415] FINISH, SetVdsStatusVDSCommand, log id: 5db39b17 >2013-03-08 12:31:06,363 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [16fa6415] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 12:31:06,385 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [2090fa30] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:31:06,388 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [27a5cc86] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:31:06,389 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-87) [27a5cc86] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 12:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) [128d0b36] Autorecovering 1 hosts >2013-03-08 12:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) [128d0b36] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-62) [4048fca7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-62) [4048fca7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-62) [4048fca7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 17985f23 >2013-03-08 12:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-62) [4048fca7] FINISH, SetVdsStatusVDSCommand, log id: 17985f23 >2013-03-08 12:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-62) [4048fca7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2c397dba >2013-03-08 12:35:00,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-62) [4048fca7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2150fa43 >2013-03-08 12:35:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-62) [4048fca7] FINISH, GetHardwareInfoVDSCommand, log id: 2150fa43 >2013-03-08 12:35:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-62) [706a39d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:35:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-62) [7f6fd8d9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:35:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-62) [7f6fd8d9] FINISH, ActivateVdsVDSCommand, log id: 2c397dba >2013-03-08 12:35:00,219 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-62) [7f6fd8d9] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:35:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-62) [7f6fd8d9] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 12:35:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) [7f6fd8d9] Autorecovering 1 storage domains >2013-03-08 12:35:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) [7f6fd8d9] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 12:35:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-62) [240ca988] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:35:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-62) [240ca988] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:35 PM >2013-03-08 12:35:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-62) [240ca988] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:35 PM >2013-03-08 12:35:00,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [1e53741e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5a374e73 >2013-03-08 12:35:00,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [1e53741e] FINISH, GetHardwareInfoVDSCommand, log id: 5a374e73 >2013-03-08 12:35:00,637 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-74) [25966068] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 12:35:00,640 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-74) [311f9827] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 12:35:00,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7b90eb76 >2013-03-08 12:36:04,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 12:36:04,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:36:04,925 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 12:36:04,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 7b90eb76 >2013-03-08 12:36:04,928 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [4f8001d1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:36:04,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [4f8001d1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3631594c >2013-03-08 12:36:04,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [4f8001d1] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 12:36:04,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [4f8001d1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3631594c >2013-03-08 12:36:04,945 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [4f8001d1] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 12:36:04,945 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [311f9827] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 12:36:04,954 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-74) [246fcf42] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:36:04,956 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [246fcf42] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: fabdd56 >2013-03-08 12:36:04,957 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [246fcf42] FINISH, SetVdsStatusVDSCommand, log id: fabdd56 >2013-03-08 12:36:04,963 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-74) [246fcf42] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 12:36:04,983 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-74) [16aaa9f3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:36:04,986 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-74) [30710378] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:36:04,987 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-74) [30710378] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 12:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [7a1f987e] Autorecovering 1 hosts >2013-03-08 12:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [7a1f987e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [6b6742ba] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [6b6742ba] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [6b6742ba] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 673a5bf0 >2013-03-08 12:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [6b6742ba] FINISH, SetVdsStatusVDSCommand, log id: 673a5bf0 >2013-03-08 12:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [6b6742ba] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 50ce1bbb >2013-03-08 12:40:00,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [6b6742ba] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4a9c8df1 >2013-03-08 12:40:00,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [6b6742ba] FINISH, GetHardwareInfoVDSCommand, log id: 4a9c8df1 >2013-03-08 12:40:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-63) [65609452] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:40:00,263 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-63) [4f03d6d6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:40:00,264 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [4f03d6d6] FINISH, ActivateVdsVDSCommand, log id: 50ce1bbb >2013-03-08 12:40:00,284 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [4f03d6d6] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:40:00,284 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-63) [4f03d6d6] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 12:40:00,286 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [4f03d6d6] Autorecovering 1 storage domains >2013-03-08 12:40:00,286 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [4f03d6d6] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 12:40:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [7c40654d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:40:00,287 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [7c40654d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:40 PM >2013-03-08 12:40:00,289 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [7c40654d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:40 PM >2013-03-08 12:40:02,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [16a2746f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 20ed684e >2013-03-08 12:40:02,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [16a2746f] FINISH, GetHardwareInfoVDSCommand, log id: 20ed684e >2013-03-08 12:40:02,238 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-84) [46ab354f] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 12:40:02,241 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-84) [813cee2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 12:40:02,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 30031e04 >2013-03-08 12:41:07,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 12:41:07,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:41:07,231 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 12:41:07,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 30031e04 >2013-03-08 12:41:07,234 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [271450fb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:41:07,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [271450fb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5e28a00b >2013-03-08 12:41:07,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [271450fb] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 12:41:07,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [271450fb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5e28a00b >2013-03-08 12:41:07,250 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [271450fb] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 12:41:07,250 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [813cee2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 12:41:07,260 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-84) [1355f98d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:41:07,261 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [1355f98d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1f2931a2 >2013-03-08 12:41:07,263 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [1355f98d] FINISH, SetVdsStatusVDSCommand, log id: 1f2931a2 >2013-03-08 12:41:07,268 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-84) [1355f98d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 12:41:07,289 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-84) [6013e9dd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:41:07,292 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-84) [3ca1cd17] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:41:07,293 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-84) [3ca1cd17] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 12:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [643024c1] Autorecovering 1 hosts >2013-03-08 12:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [643024c1] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-38) [66568511] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-38) [66568511] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [66568511] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 28ae3a15 >2013-03-08 12:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [66568511] FINISH, SetVdsStatusVDSCommand, log id: 28ae3a15 >2013-03-08 12:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-38) [66568511] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4e2a09d4 >2013-03-08 12:45:00,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [66568511] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6a3f9c40 >2013-03-08 12:45:00,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [66568511] FINISH, GetHardwareInfoVDSCommand, log id: 6a3f9c40 >2013-03-08 12:45:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-38) [5eba263a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:45:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-38) [40765f22] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:45:00,206 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-38) [40765f22] FINISH, ActivateVdsVDSCommand, log id: 4e2a09d4 >2013-03-08 12:45:00,225 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-38) [40765f22] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:45:00,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-38) [40765f22] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 12:45:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [40765f22] Autorecovering 1 storage domains >2013-03-08 12:45:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [40765f22] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 12:45:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-38) [8e7cbd9] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:45:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-38) [8e7cbd9] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:45 PM >2013-03-08 12:45:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-38) [8e7cbd9] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:45 PM >2013-03-08 12:45:01,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [171848f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4e16f70a >2013-03-08 12:45:01,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [171848f] FINISH, GetHardwareInfoVDSCommand, log id: 4e16f70a >2013-03-08 12:45:01,528 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-49) [6752230b] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 12:45:01,531 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-49) [152f8ae] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 12:45:01,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 444eff41 >2013-03-08 12:46:06,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 12:46:06,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:46:06,831 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 12:46:06,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 444eff41 >2013-03-08 12:46:06,834 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [3f78138] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:46:06,859 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [3f78138] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 20cef5ab >2013-03-08 12:46:06,859 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [3f78138] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 12:46:06,859 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [3f78138] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 20cef5ab >2013-03-08 12:46:06,861 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [3f78138] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 12:46:06,864 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [152f8ae] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 12:46:06,871 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-49) [1c7dcda5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:46:06,873 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [1c7dcda5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7cb7f33e >2013-03-08 12:46:06,874 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [1c7dcda5] FINISH, SetVdsStatusVDSCommand, log id: 7cb7f33e >2013-03-08 12:46:06,879 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-49) [1c7dcda5] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 12:46:06,900 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-49) [32b08754] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:46:06,902 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-49) [71263fbe] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:46:06,904 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) [71263fbe] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 12:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [66046291] Autorecovering 1 hosts >2013-03-08 12:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [66046291] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [e238651] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [e238651] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [e238651] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 242fbd9 >2013-03-08 12:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [e238651] FINISH, SetVdsStatusVDSCommand, log id: 242fbd9 >2013-03-08 12:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-47) [e238651] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7da72843 >2013-03-08 12:50:00,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [e238651] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3e07411 >2013-03-08 12:50:00,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [e238651] FINISH, GetHardwareInfoVDSCommand, log id: 3e07411 >2013-03-08 12:50:00,207 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [4dd70da1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:50:00,210 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [babbd3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:50:00,211 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-47) [babbd3] FINISH, ActivateVdsVDSCommand, log id: 7da72843 >2013-03-08 12:50:00,230 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [babbd3] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:50:00,231 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [babbd3] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 12:50:00,232 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [babbd3] Autorecovering 1 storage domains >2013-03-08 12:50:00,232 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [babbd3] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 12:50:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [57d9469d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:50:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [57d9469d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:50 PM >2013-03-08 12:50:00,236 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [57d9469d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:50 PM >2013-03-08 12:50:01,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [558864c3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 461a643a >2013-03-08 12:50:01,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [558864c3] FINISH, GetHardwareInfoVDSCommand, log id: 461a643a >2013-03-08 12:50:01,150 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-91) [314abb6] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 12:50:01,154 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-91) [168f7cb6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 12:50:01,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 81198a0 >2013-03-08 12:51:05,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 12:51:05,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:51:05,163 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 12:51:05,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 81198a0 >2013-03-08 12:51:05,167 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [526902b9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:51:05,181 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [526902b9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 583693b5 >2013-03-08 12:51:05,181 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [526902b9] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 12:51:05,181 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [526902b9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 583693b5 >2013-03-08 12:51:05,186 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [168f7cb6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 12:51:05,195 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-91) [1e528cb4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:51:05,196 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [1e528cb4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 69a9483c >2013-03-08 12:51:05,198 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [1e528cb4] FINISH, SetVdsStatusVDSCommand, log id: 69a9483c >2013-03-08 12:51:05,203 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-91) [1e528cb4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 12:51:05,224 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-91) [2c2d06bc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:51:05,226 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-91) [1ce1a6db] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:51:05,228 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-91) [1ce1a6db] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 12:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [539dc993] Autorecovering 1 hosts >2013-03-08 12:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [539dc993] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:55:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [4f3b9591] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [4f3b9591] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [4f3b9591] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 517e9963 >2013-03-08 12:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [4f3b9591] FINISH, SetVdsStatusVDSCommand, log id: 517e9963 >2013-03-08 12:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [4f3b9591] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3582bbb2 >2013-03-08 12:55:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [4f3b9591] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3f45475a >2013-03-08 12:55:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [4f3b9591] FINISH, GetHardwareInfoVDSCommand, log id: 3f45475a >2013-03-08 12:55:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-96) [2c710434] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:55:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-96) [2c6dbed1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:55:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [2c6dbed1] FINISH, ActivateVdsVDSCommand, log id: 3582bbb2 >2013-03-08 12:55:00,228 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [2c6dbed1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 12:55:00,229 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-96) [2c6dbed1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 12:55:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [2c6dbed1] Autorecovering 1 storage domains >2013-03-08 12:55:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [2c6dbed1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 12:55:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [23ffeabc] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:55:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [23ffeabc] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 12:55 PM >2013-03-08 12:55:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [23ffeabc] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 12:55 PM >2013-03-08 12:55:02,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [4ed5b2f0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 69bf6c8c >2013-03-08 12:55:02,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [4ed5b2f0] FINISH, GetHardwareInfoVDSCommand, log id: 69bf6c8c >2013-03-08 12:55:02,475 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-24) [4d041e50] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 12:55:02,479 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-24) [545b24a1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 12:55:02,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: fe4a4bd >2013-03-08 12:56:07,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 12:56:07,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 12:56:07,526 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 12:56:07,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: fe4a4bd >2013-03-08 12:56:07,530 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [7ea8c137] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 12:56:07,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [7ea8c137] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 31fce776 >2013-03-08 12:56:07,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [7ea8c137] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 12:56:07,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [7ea8c137] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 31fce776 >2013-03-08 12:56:07,546 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [7ea8c137] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 12:56:07,547 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [545b24a1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 12:56:07,573 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-24) [8d71183] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:56:07,575 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [8d71183] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 17bf077a >2013-03-08 12:56:07,576 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [8d71183] FINISH, SetVdsStatusVDSCommand, log id: 17bf077a >2013-03-08 12:56:07,582 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-24) [8d71183] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 12:56:07,603 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-24) [7455d90f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:56:07,606 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-24) [3113df82] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 12:56:07,607 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-24) [3113df82] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 13:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-37) [2aa04f43] Autorecovering 1 hosts >2013-03-08 13:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-37) [2aa04f43] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-37) [268a18bf] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-37) [268a18bf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [268a18bf] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4c42187d >2013-03-08 13:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [268a18bf] FINISH, SetVdsStatusVDSCommand, log id: 4c42187d >2013-03-08 13:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-37) [268a18bf] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 55158146 >2013-03-08 13:00:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [268a18bf] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6ad2707d >2013-03-08 13:00:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [268a18bf] FINISH, GetHardwareInfoVDSCommand, log id: 6ad2707d >2013-03-08 13:00:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-37) [7aa672ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:00:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-37) [5b097176] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:00:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-37) [5b097176] FINISH, ActivateVdsVDSCommand, log id: 55158146 >2013-03-08 13:00:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-37) [5b097176] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:00:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-37) [5b097176] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 13:00:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-37) [5b097176] Autorecovering 1 storage domains >2013-03-08 13:00:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-37) [5b097176] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 13:00:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-37) [6a5a04e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:00:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-37) [6a5a04e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:00 PM >2013-03-08 13:00:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-37) [6a5a04e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:00 PM >2013-03-08 13:00:01,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [7e57290] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4ac1d893 >2013-03-08 13:00:01,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [7e57290] FINISH, GetHardwareInfoVDSCommand, log id: 4ac1d893 >2013-03-08 13:00:01,853 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-13) [25603021] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 13:00:01,857 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-13) [185959a3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 13:00:01,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2c01eee5 >2013-03-08 13:01:06,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 13:01:06,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:01:06,888 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 13:01:06,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 2c01eee5 >2013-03-08 13:01:06,891 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [3dd1d43f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:01:06,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [3dd1d43f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 62736ddf >2013-03-08 13:01:06,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [3dd1d43f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 13:01:06,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [3dd1d43f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 62736ddf >2013-03-08 13:01:06,909 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [3dd1d43f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 13:01:06,910 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [185959a3] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 13:01:06,919 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-13) [2757a4be] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:01:06,921 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [2757a4be] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 73d28b04 >2013-03-08 13:01:06,922 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [2757a4be] FINISH, SetVdsStatusVDSCommand, log id: 73d28b04 >2013-03-08 13:01:06,927 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-13) [2757a4be] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 13:01:06,952 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-13) [c849175] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:01:06,954 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-13) [1456f04f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:01:06,956 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-13) [1456f04f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 13:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [12425f9d] Autorecovering 1 hosts >2013-03-08 13:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [12425f9d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [658b24e2] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [658b24e2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [658b24e2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 786b3ee1 >2013-03-08 13:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [658b24e2] FINISH, SetVdsStatusVDSCommand, log id: 786b3ee1 >2013-03-08 13:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-64) [658b24e2] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6d36fa14 >2013-03-08 13:05:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [658b24e2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 316dd8e3 >2013-03-08 13:05:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [658b24e2] FINISH, GetHardwareInfoVDSCommand, log id: 316dd8e3 >2013-03-08 13:05:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-64) [329de624] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:05:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-64) [47e05d31] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:05:00,203 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-64) [47e05d31] FINISH, ActivateVdsVDSCommand, log id: 6d36fa14 >2013-03-08 13:05:00,222 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [47e05d31] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:05:00,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-64) [47e05d31] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 13:05:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [47e05d31] Autorecovering 1 storage domains >2013-03-08 13:05:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [47e05d31] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 13:05:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [e8ee453] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:05:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [e8ee453] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:05 PM >2013-03-08 13:05:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [e8ee453] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:05 PM >2013-03-08 13:05:01,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [3245cecf] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7aac43a7 >2013-03-08 13:05:01,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [3245cecf] FINISH, GetHardwareInfoVDSCommand, log id: 7aac43a7 >2013-03-08 13:05:01,205 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-86) [771e4461] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 13:05:01,208 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-86) [2e996] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 13:05:01,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 88afbd0 >2013-03-08 13:06:04,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 13:06:04,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:06:04,211 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 13:06:04,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 88afbd0 >2013-03-08 13:06:04,214 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [48d6666e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:06:04,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [48d6666e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 62922143 >2013-03-08 13:06:04,229 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [48d6666e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 13:06:04,229 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [48d6666e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 62922143 >2013-03-08 13:06:04,231 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [48d6666e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 13:06:04,231 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [2e996] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 13:06:04,241 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-86) [1b8a2499] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:06:04,242 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [1b8a2499] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5be68be4 >2013-03-08 13:06:04,243 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [1b8a2499] FINISH, SetVdsStatusVDSCommand, log id: 5be68be4 >2013-03-08 13:06:04,249 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-86) [1b8a2499] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 13:06:04,270 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-86) [499100f6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:06:04,272 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-86) [5c61f6bb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:06:04,274 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-86) [5c61f6bb] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 13:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [47d9290b] Autorecovering 1 hosts >2013-03-08 13:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [47d9290b] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [6df91732] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [6df91732] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [6df91732] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: b569a85 >2013-03-08 13:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [6df91732] FINISH, SetVdsStatusVDSCommand, log id: b569a85 >2013-03-08 13:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-19) [6df91732] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 51b9f05d >2013-03-08 13:10:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [6df91732] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 58eb6954 >2013-03-08 13:10:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [6df91732] FINISH, GetHardwareInfoVDSCommand, log id: 58eb6954 >2013-03-08 13:10:00,258 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-19) [726c7cea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:10:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-19) [ceb2773] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:10:00,261 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-19) [ceb2773] FINISH, ActivateVdsVDSCommand, log id: 51b9f05d >2013-03-08 13:10:00,281 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [ceb2773] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:10:00,282 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-19) [ceb2773] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 13:10:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [ceb2773] Autorecovering 1 storage domains >2013-03-08 13:10:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [ceb2773] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 13:10:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [a9105b1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:10:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [a9105b1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:10 PM >2013-03-08 13:10:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [a9105b1] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:10 PM >2013-03-08 13:10:01,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [7b066bd7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4a88332c >2013-03-08 13:10:01,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [7b066bd7] FINISH, GetHardwareInfoVDSCommand, log id: 4a88332c >2013-03-08 13:10:01,544 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-43) [580aeb72] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 13:10:01,548 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-43) [60f164bd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 13:10:01,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7b90c2fb >2013-03-08 13:11:06,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 13:11:06,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:11:06,591 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 13:11:06,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 7b90c2fb >2013-03-08 13:11:06,595 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [294b5f24] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:11:06,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [294b5f24] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7dd14de3 >2013-03-08 13:11:06,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [294b5f24] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 13:11:06,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [294b5f24] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7dd14de3 >2013-03-08 13:11:06,612 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [294b5f24] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 13:11:06,613 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [60f164bd] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 13:11:06,622 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-43) [6734a546] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:11:06,624 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [6734a546] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7c1f1606 >2013-03-08 13:11:06,625 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [6734a546] FINISH, SetVdsStatusVDSCommand, log id: 7c1f1606 >2013-03-08 13:11:06,631 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [6734a546] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 13:11:06,652 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [347ee153] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:11:06,655 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [15dee6d7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:11:06,656 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-43) [15dee6d7] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 13:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [6e14840e] Autorecovering 1 hosts >2013-03-08 13:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [6e14840e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [54212165] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [54212165] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [54212165] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 22e71c2c >2013-03-08 13:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [54212165] FINISH, SetVdsStatusVDSCommand, log id: 22e71c2c >2013-03-08 13:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [54212165] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7adf3228 >2013-03-08 13:15:00,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [54212165] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6e19ea2 >2013-03-08 13:15:00,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [54212165] FINISH, GetHardwareInfoVDSCommand, log id: 6e19ea2 >2013-03-08 13:15:00,262 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-16) [3a8b97c5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:15:00,264 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-16) [711658ed] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:15:00,266 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [711658ed] FINISH, ActivateVdsVDSCommand, log id: 7adf3228 >2013-03-08 13:15:00,285 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [711658ed] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:15:00,286 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-16) [711658ed] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 13:15:00,287 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [711658ed] Autorecovering 1 storage domains >2013-03-08 13:15:00,287 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [711658ed] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 13:15:00,288 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [27c46bf3] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:15:00,288 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [27c46bf3] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:15 PM >2013-03-08 13:15:00,291 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [27c46bf3] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:15 PM >2013-03-08 13:15:00,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [6c97e9a6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 376cb3b >2013-03-08 13:15:00,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [6c97e9a6] FINISH, GetHardwareInfoVDSCommand, log id: 376cb3b >2013-03-08 13:15:00,901 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-1) [4c2c6221] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 13:15:00,905 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-1) [59051502] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 13:15:00,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2de278c7 >2013-03-08 13:16:04,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 13:16:04,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:16:04,900 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 13:16:04,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 2de278c7 >2013-03-08 13:16:04,903 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [58d8c686] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:16:04,917 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [58d8c686] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3b836630 >2013-03-08 13:16:04,917 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [58d8c686] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 13:16:04,917 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [58d8c686] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3b836630 >2013-03-08 13:16:04,919 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [58d8c686] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 13:16:04,919 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [59051502] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 13:16:04,930 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-1) [405bd8f2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:16:04,931 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [405bd8f2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 50b8f48b >2013-03-08 13:16:04,932 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [405bd8f2] FINISH, SetVdsStatusVDSCommand, log id: 50b8f48b >2013-03-08 13:16:04,938 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-1) [405bd8f2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 13:16:04,963 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-1) [1e1f49e0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:16:04,965 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-1) [6be148e9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:16:04,966 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-1) [6be148e9] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 13:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [e5d01fb] Autorecovering 1 hosts >2013-03-08 13:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [e5d01fb] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [1c37cc82] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [1c37cc82] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [1c37cc82] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2f2f2e1a >2013-03-08 13:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [1c37cc82] FINISH, SetVdsStatusVDSCommand, log id: 2f2f2e1a >2013-03-08 13:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [1c37cc82] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: acdb748 >2013-03-08 13:20:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [1c37cc82] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5e1daed9 >2013-03-08 13:20:00,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [1c37cc82] FINISH, GetHardwareInfoVDSCommand, log id: 5e1daed9 >2013-03-08 13:20:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [45e81cf5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:20:00,208 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [35d13e0a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:20:00,209 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [35d13e0a] FINISH, ActivateVdsVDSCommand, log id: acdb748 >2013-03-08 13:20:00,228 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [35d13e0a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:20:00,229 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-8) [35d13e0a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 13:20:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [35d13e0a] Autorecovering 1 storage domains >2013-03-08 13:20:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [35d13e0a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 13:20:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [70a3eb0a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:20:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [70a3eb0a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:20 PM >2013-03-08 13:20:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [70a3eb0a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:20 PM >2013-03-08 13:20:02,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [d1e7f84] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 122eb4c8 >2013-03-08 13:20:02,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [d1e7f84] FINISH, GetHardwareInfoVDSCommand, log id: 122eb4c8 >2013-03-08 13:20:02,213 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-34) [202aedc3] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 13:20:02,216 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-34) [1058a86b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 13:20:02,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 747f2b70 >2013-03-08 13:21:08,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 13:21:08,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:21:08,216 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 13:21:08,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 747f2b70 >2013-03-08 13:21:08,220 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [553985de] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:21:08,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [553985de] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6ce367f1 >2013-03-08 13:21:08,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [553985de] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 13:21:08,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [553985de] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6ce367f1 >2013-03-08 13:21:08,236 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [553985de] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 13:21:08,236 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [1058a86b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 13:21:08,246 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-34) [1711367a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:21:08,248 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [1711367a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 791871be >2013-03-08 13:21:08,249 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [1711367a] FINISH, SetVdsStatusVDSCommand, log id: 791871be >2013-03-08 13:21:08,255 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-34) [1711367a] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 13:21:08,275 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-34) [79a8b31f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:21:08,278 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-34) [4989b114] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:21:08,279 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) [4989b114] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 13:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [5f361390] Autorecovering 1 hosts >2013-03-08 13:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [5f361390] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [5e94cf36] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [5e94cf36] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:25:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [5e94cf36] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: fd11980 >2013-03-08 13:25:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [5e94cf36] FINISH, SetVdsStatusVDSCommand, log id: fd11980 >2013-03-08 13:25:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-44) [5e94cf36] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 43596190 >2013-03-08 13:25:00,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [5e94cf36] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6809ebb9 >2013-03-08 13:25:00,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [5e94cf36] FINISH, GetHardwareInfoVDSCommand, log id: 6809ebb9 >2013-03-08 13:25:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-44) [a0cbf0f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:25:00,207 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-44) [77eafeeb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:25:00,209 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-44) [77eafeeb] FINISH, ActivateVdsVDSCommand, log id: 43596190 >2013-03-08 13:25:00,228 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [77eafeeb] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:25:00,229 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-44) [77eafeeb] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 13:25:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [77eafeeb] Autorecovering 1 storage domains >2013-03-08 13:25:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [77eafeeb] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 13:25:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [37913a0f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:25:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [37913a0f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:25 PM >2013-03-08 13:25:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [37913a0f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:25 PM >2013-03-08 13:25:02,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [290d4943] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 172a4c56 >2013-03-08 13:25:02,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [290d4943] FINISH, GetHardwareInfoVDSCommand, log id: 172a4c56 >2013-03-08 13:25:02,523 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-31) [77612aee] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 13:25:02,527 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-31) [20848191] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 13:25:02,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 35ec1d19 >2013-03-08 13:26:06,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 13:26:06,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:26:06,566 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 13:26:06,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 35ec1d19 >2013-03-08 13:26:06,570 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [6f4ba880] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:26:06,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [6f4ba880] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2dd1ddde >2013-03-08 13:26:06,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [6f4ba880] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 13:26:06,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [6f4ba880] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2dd1ddde >2013-03-08 13:26:06,586 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [6f4ba880] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 13:26:06,586 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [20848191] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 13:26:06,599 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-31) [28ea8a9b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:26:06,600 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [28ea8a9b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7fd70aec >2013-03-08 13:26:06,601 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [28ea8a9b] FINISH, SetVdsStatusVDSCommand, log id: 7fd70aec >2013-03-08 13:26:06,607 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-31) [28ea8a9b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 13:26:06,628 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-31) [58874f2a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:26:06,631 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-31) [310fbd06] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:26:06,632 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-31) [310fbd06] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 13:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [77295e42] Autorecovering 1 hosts >2013-03-08 13:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [77295e42] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-25) [2a8aec04] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-25) [2a8aec04] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [2a8aec04] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 17bb02fd >2013-03-08 13:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [2a8aec04] FINISH, SetVdsStatusVDSCommand, log id: 17bb02fd >2013-03-08 13:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-25) [2a8aec04] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6aad1e7d >2013-03-08 13:30:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [2a8aec04] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: f292db2 >2013-03-08 13:30:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [2a8aec04] FINISH, GetHardwareInfoVDSCommand, log id: f292db2 >2013-03-08 13:30:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-25) [261b13d0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:30:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-25) [448372e1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:30:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-25) [448372e1] FINISH, ActivateVdsVDSCommand, log id: 6aad1e7d >2013-03-08 13:30:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-25) [448372e1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:30:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-25) [448372e1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 13:30:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [448372e1] Autorecovering 1 storage domains >2013-03-08 13:30:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [448372e1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 13:30:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-25) [fb36739] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:30:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-25) [fb36739] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:30 PM >2013-03-08 13:30:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-25) [fb36739] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:30 PM >2013-03-08 13:30:00,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [5b81993a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4966e87f >2013-03-08 13:30:00,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [5b81993a] FINISH, GetHardwareInfoVDSCommand, log id: 4966e87f >2013-03-08 13:30:00,875 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-73) [5b33b9a7] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 13:30:00,878 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-73) [7bfe70b2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 13:30:00,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 438a79c0 >2013-03-08 13:30:33,742 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-56) [d1e2a77] Start refreshing all users data >2013-03-08 13:30:34,688 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-72) [497e931f] Attempting to update VMs/Templates Ovf. >2013-03-08 13:30:34,689 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-72) [497e931f] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 13:30:34,689 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-72) [497e931f] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 13:30:34,690 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-72) [497e931f] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 13:30:34,690 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-72) [497e931f] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 13:30:34,690 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-72) [497e931f] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 13:30:34,691 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-72) [497e931f] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 13:31:05,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 13:31:05,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:31:05,888 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 13:31:05,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 438a79c0 >2013-03-08 13:31:05,891 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [2f45294a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:31:05,905 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [2f45294a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 629a53f9 >2013-03-08 13:31:05,905 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [2f45294a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 13:31:05,905 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [2f45294a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 629a53f9 >2013-03-08 13:31:05,907 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [2f45294a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 13:31:05,908 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [7bfe70b2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 13:31:05,922 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-73) [1e77a6c3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:31:05,923 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [1e77a6c3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 16b405ca >2013-03-08 13:31:05,925 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [1e77a6c3] FINISH, SetVdsStatusVDSCommand, log id: 16b405ca >2013-03-08 13:31:05,931 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-73) [1e77a6c3] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 13:31:05,955 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-73) [42b86b08] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:31:05,957 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-73) [10e3e76a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:31:05,959 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-73) [10e3e76a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 13:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [5bb0d0b6] Autorecovering 1 hosts >2013-03-08 13:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [5bb0d0b6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [224718eb] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [224718eb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [224718eb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4f84d7d8 >2013-03-08 13:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [224718eb] FINISH, SetVdsStatusVDSCommand, log id: 4f84d7d8 >2013-03-08 13:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-93) [224718eb] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 224e4afc >2013-03-08 13:35:00,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [224718eb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1dbabf26 >2013-03-08 13:35:00,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [224718eb] FINISH, GetHardwareInfoVDSCommand, log id: 1dbabf26 >2013-03-08 13:35:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [9192871] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:35:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [67f93fba] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:35:00,206 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-93) [67f93fba] FINISH, ActivateVdsVDSCommand, log id: 224e4afc >2013-03-08 13:35:00,226 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [67f93fba] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:35:00,227 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-93) [67f93fba] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 13:35:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [67f93fba] Autorecovering 1 storage domains >2013-03-08 13:35:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [67f93fba] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 13:35:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [cf9e7f6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:35:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [cf9e7f6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:35 PM >2013-03-08 13:35:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [cf9e7f6] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:35 PM >2013-03-08 13:35:00,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-3) [445f1240] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5db27a75 >2013-03-08 13:35:00,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-3) [445f1240] FINISH, GetHardwareInfoVDSCommand, log id: 5db27a75 >2013-03-08 13:35:00,304 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-3) [4819244c] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 13:35:00,308 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-3) [7c7ceafe] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 13:35:00,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6968fab6 >2013-03-08 13:36:04,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 13:36:04,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:36:04,619 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 13:36:04,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 6968fab6 >2013-03-08 13:36:04,623 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [6395a731] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:36:04,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [6395a731] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4559bfd6 >2013-03-08 13:36:04,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [6395a731] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 13:36:04,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [6395a731] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4559bfd6 >2013-03-08 13:36:04,639 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [6395a731] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 13:36:04,639 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [7c7ceafe] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 13:36:04,649 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-3) [49355874] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:36:04,650 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-3) [49355874] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 45e7d68e >2013-03-08 13:36:04,652 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-3) [49355874] FINISH, SetVdsStatusVDSCommand, log id: 45e7d68e >2013-03-08 13:36:04,657 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) [49355874] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 13:36:04,678 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-3) [384386aa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:36:04,680 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-3) [e986c15] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:36:04,682 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-3) [e986c15] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 13:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [728b1608] Autorecovering 1 hosts >2013-03-08 13:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [728b1608] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-51) [4737965] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-51) [4737965] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [4737965] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 63253a56 >2013-03-08 13:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [4737965] FINISH, SetVdsStatusVDSCommand, log id: 63253a56 >2013-03-08 13:40:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-51) [4737965] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6c3e18d3 >2013-03-08 13:40:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [4737965] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1cd9a0db >2013-03-08 13:40:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [4737965] FINISH, GetHardwareInfoVDSCommand, log id: 1cd9a0db >2013-03-08 13:40:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-51) [40f35c6c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:40:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-51) [353d5a23] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:40:00,203 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-51) [353d5a23] FINISH, ActivateVdsVDSCommand, log id: 6c3e18d3 >2013-03-08 13:40:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-51) [353d5a23] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:40:00,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-51) [353d5a23] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 13:40:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [353d5a23] Autorecovering 1 storage domains >2013-03-08 13:40:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [353d5a23] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 13:40:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-51) [6fd08d94] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:40:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-51) [6fd08d94] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:40 PM >2013-03-08 13:40:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-51) [6fd08d94] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:40 PM >2013-03-08 13:40:01,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [27a5cc86] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 635fcd6c >2013-03-08 13:40:01,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [27a5cc86] FINISH, GetHardwareInfoVDSCommand, log id: 635fcd6c >2013-03-08 13:40:01,929 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-87) [6e4675e0] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 13:40:01,932 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-87) [827b660] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 13:40:01,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6d521097 >2013-03-08 13:41:04,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 13:41:04,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:41:04,920 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 13:41:04,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 6d521097 >2013-03-08 13:41:04,924 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [10d260fb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:41:04,938 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [10d260fb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 563ac3 >2013-03-08 13:41:04,938 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [10d260fb] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 13:41:04,938 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [10d260fb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 563ac3 >2013-03-08 13:41:04,940 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [10d260fb] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 13:41:04,941 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [827b660] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 13:41:04,950 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-87) [35dede16] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:41:04,952 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [35dede16] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4d5d13a6 >2013-03-08 13:41:04,953 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [35dede16] FINISH, SetVdsStatusVDSCommand, log id: 4d5d13a6 >2013-03-08 13:41:04,959 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [35dede16] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 13:41:04,981 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [4098170e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:41:04,983 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [598f926d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:41:04,984 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-87) [598f926d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 13:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [452d305b] Autorecovering 1 hosts >2013-03-08 13:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [452d305b] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [d5545e0] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [d5545e0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [d5545e0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 20c56f48 >2013-03-08 13:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [d5545e0] FINISH, SetVdsStatusVDSCommand, log id: 20c56f48 >2013-03-08 13:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-9) [d5545e0] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7ae3793 >2013-03-08 13:45:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [d5545e0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1d65b9e3 >2013-03-08 13:45:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [d5545e0] FINISH, GetHardwareInfoVDSCommand, log id: 1d65b9e3 >2013-03-08 13:45:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-9) [39f45190] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:45:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-9) [2f4b5bf3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:45:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-9) [2f4b5bf3] FINISH, ActivateVdsVDSCommand, log id: 7ae3793 >2013-03-08 13:45:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [2f4b5bf3] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:45:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-9) [2f4b5bf3] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 13:45:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [2f4b5bf3] Autorecovering 1 storage domains >2013-03-08 13:45:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [2f4b5bf3] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 13:45:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [207c5a5b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:45:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [207c5a5b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:45 PM >2013-03-08 13:45:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [207c5a5b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:45 PM >2013-03-08 13:45:02,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [eeec576] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 190ee896 >2013-03-08 13:45:02,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [eeec576] FINISH, GetHardwareInfoVDSCommand, log id: 190ee896 >2013-03-08 13:45:02,231 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-68) [85434ec] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 13:45:02,234 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-68) [2fb249ed] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 13:45:02,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7e709eda >2013-03-08 13:46:07,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 13:46:07,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:46:07,221 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 13:46:07,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 7e709eda >2013-03-08 13:46:07,224 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [7dfdd6eb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:46:07,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [7dfdd6eb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 440dd5ec >2013-03-08 13:46:07,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [7dfdd6eb] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 13:46:07,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [7dfdd6eb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 440dd5ec >2013-03-08 13:46:07,240 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [7dfdd6eb] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 13:46:07,241 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [2fb249ed] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 13:46:07,250 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-68) [ed461fa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:46:07,252 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [ed461fa] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 74fcb8fd >2013-03-08 13:46:07,253 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [ed461fa] FINISH, SetVdsStatusVDSCommand, log id: 74fcb8fd >2013-03-08 13:46:07,258 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-68) [ed461fa] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 13:46:07,279 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-68) [68252ac2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:46:07,281 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-68) [1902f6b0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:46:07,283 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-68) [1902f6b0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 13:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [6ab4d6b6] Autorecovering 1 hosts >2013-03-08 13:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [6ab4d6b6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [1b02560e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [1b02560e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [1b02560e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: c5f7721 >2013-03-08 13:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [1b02560e] FINISH, SetVdsStatusVDSCommand, log id: c5f7721 >2013-03-08 13:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [1b02560e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: cefbf >2013-03-08 13:50:00,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [1b02560e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5b41e319 >2013-03-08 13:50:00,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [1b02560e] FINISH, GetHardwareInfoVDSCommand, log id: 5b41e319 >2013-03-08 13:50:00,318 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [71e6d444] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:50:00,320 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [3168ead8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:50:00,321 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [3168ead8] FINISH, ActivateVdsVDSCommand, log id: cefbf >2013-03-08 13:50:00,341 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [3168ead8] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:50:00,342 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [3168ead8] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 13:50:00,343 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [3168ead8] Autorecovering 1 storage domains >2013-03-08 13:50:00,343 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [3168ead8] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 13:50:00,343 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [2a4ad98c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:50:00,344 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [2a4ad98c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:50 PM >2013-03-08 13:50:00,346 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [2a4ad98c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:50 PM >2013-03-08 13:50:01,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [45f5e7be] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6d867d3a >2013-03-08 13:50:01,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [45f5e7be] FINISH, GetHardwareInfoVDSCommand, log id: 6d867d3a >2013-03-08 13:50:01,554 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-94) [351310f5] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 13:50:01,557 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-94) [65e48bd5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 13:50:01,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 36cf7c56 >2013-03-08 13:51:05,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 13:51:05,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:51:05,539 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 13:51:05,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 36cf7c56 >2013-03-08 13:51:05,543 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [400ef3c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:51:05,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [400ef3c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 14956291 >2013-03-08 13:51:05,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [400ef3c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 13:51:05,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [400ef3c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 14956291 >2013-03-08 13:51:05,562 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [65e48bd5] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 13:51:05,572 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-94) [fede841] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:51:05,574 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [fede841] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 556ae84a >2013-03-08 13:51:05,575 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [fede841] FINISH, SetVdsStatusVDSCommand, log id: 556ae84a >2013-03-08 13:51:05,581 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-94) [fede841] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 13:51:05,601 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [711fd30e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:51:05,603 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [1e4d912e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:51:05,605 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-94) [1e4d912e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 13:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [7882a1df] Autorecovering 1 hosts >2013-03-08 13:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [7882a1df] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [2c012cc0] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [2c012cc0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [2c012cc0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 29fdab1c >2013-03-08 13:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [2c012cc0] FINISH, SetVdsStatusVDSCommand, log id: 29fdab1c >2013-03-08 13:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-46) [2c012cc0] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 14c792bb >2013-03-08 13:55:00,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [2c012cc0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7ba37ddd >2013-03-08 13:55:00,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [2c012cc0] FINISH, GetHardwareInfoVDSCommand, log id: 7ba37ddd >2013-03-08 13:55:00,540 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-46) [3463a31] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:55:00,543 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-46) [6cf3acef] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:55:00,544 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-46) [6cf3acef] FINISH, ActivateVdsVDSCommand, log id: 14c792bb >2013-03-08 13:55:00,563 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [6cf3acef] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 13:55:00,564 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-46) [6cf3acef] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 13:55:00,565 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [6cf3acef] Autorecovering 1 storage domains >2013-03-08 13:55:00,566 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [6cf3acef] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 13:55:00,566 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [688afdbc] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:55:00,566 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [688afdbc] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 1:55 PM >2013-03-08 13:55:00,569 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [688afdbc] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 1:55 PM >2013-03-08 13:55:02,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [71263fbe] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5ef8400f >2013-03-08 13:55:02,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [71263fbe] FINISH, GetHardwareInfoVDSCommand, log id: 5ef8400f >2013-03-08 13:55:02,852 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-49) [3e93bd76] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 13:55:02,855 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-49) [74eb00f6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 13:55:02,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 117bd7b4 >2013-03-08 13:56:08,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 13:56:08,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 13:56:08,890 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 13:56:08,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 117bd7b4 >2013-03-08 13:56:08,894 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [60741810] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 13:56:08,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [60741810] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: ad6fbc9 >2013-03-08 13:56:08,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [60741810] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 13:56:08,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [60741810] FINISH, MarkPoolInReconstructModeVDSCommand, log id: ad6fbc9 >2013-03-08 13:56:08,910 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [60741810] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 13:56:08,910 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [74eb00f6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 13:56:08,920 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-49) [726aec87] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:56:08,921 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [726aec87] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 54898128 >2013-03-08 13:56:08,923 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [726aec87] FINISH, SetVdsStatusVDSCommand, log id: 54898128 >2013-03-08 13:56:08,928 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-49) [726aec87] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 13:56:08,950 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-49) [64f28fdb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:56:08,953 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-49) [5505c21c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 13:56:08,954 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) [5505c21c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 14:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [1456f04f] Autorecovering 1 hosts >2013-03-08 14:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [1456f04f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [5626d7aa] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [5626d7aa] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [5626d7aa] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5bd0319f >2013-03-08 14:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [5626d7aa] FINISH, SetVdsStatusVDSCommand, log id: 5bd0319f >2013-03-08 14:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-13) [5626d7aa] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3b5c1928 >2013-03-08 14:00:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [5626d7aa] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 765ba7a7 >2013-03-08 14:00:00,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [5626d7aa] FINISH, GetHardwareInfoVDSCommand, log id: 765ba7a7 >2013-03-08 14:00:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-13) [4d385b28] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:00:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-13) [26262d6a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:00:00,093 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-13) [26262d6a] FINISH, ActivateVdsVDSCommand, log id: 3b5c1928 >2013-03-08 14:00:00,118 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [26262d6a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:00:00,119 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-13) [26262d6a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 14:00:00,120 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [26262d6a] Autorecovering 1 storage domains >2013-03-08 14:00:00,120 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [26262d6a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 14:00:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [20f4a0a1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:00:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [20f4a0a1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:00 PM >2013-03-08 14:00:00,124 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [20f4a0a1] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:00 PM >2013-03-08 14:00:00,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-62) [240ca988] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 21ae0d7d >2013-03-08 14:00:00,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-62) [240ca988] FINISH, GetHardwareInfoVDSCommand, log id: 21ae0d7d >2013-03-08 14:00:00,203 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-62) [6f3529f6] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 14:00:00,206 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-62) [705a622e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 14:00:00,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1f205225 >2013-03-08 14:01:06,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 14:01:06,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:01:06,211 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 14:01:06,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 1f205225 >2013-03-08 14:01:06,214 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [5bbcf93] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:01:06,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5bbcf93] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 47c8e424 >2013-03-08 14:01:06,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [5bbcf93] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 14:01:06,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5bbcf93] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 47c8e424 >2013-03-08 14:01:06,230 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [5bbcf93] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 14:01:06,230 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [705a622e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 14:01:06,240 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-62) [54954024] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:01:06,242 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-62) [54954024] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2126281 >2013-03-08 14:01:06,243 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-62) [54954024] FINISH, SetVdsStatusVDSCommand, log id: 2126281 >2013-03-08 14:01:06,249 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-62) [54954024] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 14:01:06,270 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-62) [13d67f9a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:01:06,273 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-62) [1cbb8e9a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:01:06,274 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-62) [1cbb8e9a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 14:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [21f8e886] Autorecovering 1 hosts >2013-03-08 14:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [21f8e886] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [34f20e0a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [34f20e0a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-29) [34f20e0a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 358ad5d2 >2013-03-08 14:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-29) [34f20e0a] FINISH, SetVdsStatusVDSCommand, log id: 358ad5d2 >2013-03-08 14:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-29) [34f20e0a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 78f7eea3 >2013-03-08 14:05:00,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-29) [34f20e0a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2958281a >2013-03-08 14:05:00,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-29) [34f20e0a] FINISH, GetHardwareInfoVDSCommand, log id: 2958281a >2013-03-08 14:05:00,185 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-29) [3f36a131] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:05:00,188 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-29) [4d812cc1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:05:00,189 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-29) [4d812cc1] FINISH, ActivateVdsVDSCommand, log id: 78f7eea3 >2013-03-08 14:05:00,208 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [4d812cc1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:05:00,209 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-29) [4d812cc1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 14:05:00,211 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [4d812cc1] Autorecovering 1 storage domains >2013-03-08 14:05:00,211 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [4d812cc1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 14:05:00,211 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [2bb2d7fd] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:05:00,211 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [2bb2d7fd] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:05 PM >2013-03-08 14:05:00,214 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [2bb2d7fd] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:05 PM >2013-03-08 14:05:00,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [7c40654d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2c739d37 >2013-03-08 14:05:00,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [7c40654d] FINISH, GetHardwareInfoVDSCommand, log id: 2c739d37 >2013-03-08 14:05:00,514 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-63) [6c596b] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 14:05:00,517 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-63) [51f2e0a0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 14:05:00,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 354d84f0 >2013-03-08 14:06:06,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 14:06:06,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:06:06,544 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 14:06:06,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 354d84f0 >2013-03-08 14:06:06,548 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [54427841] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:06:06,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [54427841] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5f3aa32e >2013-03-08 14:06:06,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [54427841] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 14:06:06,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [54427841] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5f3aa32e >2013-03-08 14:06:06,564 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [54427841] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 14:06:06,564 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [51f2e0a0] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 14:06:06,573 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-63) [6a998235] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:06:06,575 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [6a998235] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 56c9c1db >2013-03-08 14:06:06,576 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [6a998235] FINISH, SetVdsStatusVDSCommand, log id: 56c9c1db >2013-03-08 14:06:06,590 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-63) [6a998235] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 14:06:06,611 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-63) [64b4961c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:06:06,613 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-63) [7819a10c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:06:06,615 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-63) [7819a10c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 14:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [15dee6d7] Autorecovering 1 hosts >2013-03-08 14:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [15dee6d7] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [1025d70f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [1025d70f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [1025d70f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7a6a24f6 >2013-03-08 14:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [1025d70f] FINISH, SetVdsStatusVDSCommand, log id: 7a6a24f6 >2013-03-08 14:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [1025d70f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7d36ec6f >2013-03-08 14:10:00,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [1025d70f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5df2cac0 >2013-03-08 14:10:00,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [1025d70f] FINISH, GetHardwareInfoVDSCommand, log id: 5df2cac0 >2013-03-08 14:10:00,602 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [4571f334] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:10:00,604 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [3f479b36] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:10:00,606 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [3f479b36] FINISH, ActivateVdsVDSCommand, log id: 7d36ec6f >2013-03-08 14:10:00,625 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [3f479b36] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:10:00,626 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [3f479b36] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 14:10:00,627 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [3f479b36] Autorecovering 1 storage domains >2013-03-08 14:10:00,627 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [3f479b36] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 14:10:00,628 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [552c831b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:10:00,628 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [552c831b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:10 PM >2013-03-08 14:10:00,631 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [552c831b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:10 PM >2013-03-08 14:10:00,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-6) [5355f28c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 52108ba4 >2013-03-08 14:10:00,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-6) [5355f28c] FINISH, GetHardwareInfoVDSCommand, log id: 52108ba4 >2013-03-08 14:10:00,859 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-6) [45d28afc] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 14:10:00,863 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-6) [3e072b3b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 14:10:00,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 28dd2c2d >2013-03-08 14:11:05,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 14:11:05,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:11:05,906 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 14:11:05,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 28dd2c2d >2013-03-08 14:11:05,910 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [25760f36] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:11:05,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [25760f36] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 27f6c1f1 >2013-03-08 14:11:05,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [25760f36] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 14:11:05,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [25760f36] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 27f6c1f1 >2013-03-08 14:11:05,925 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [25760f36] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 14:11:05,926 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [3e072b3b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 14:11:05,935 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-6) [f42ccd0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:11:05,937 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-6) [f42ccd0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5eba2505 >2013-03-08 14:11:05,938 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-6) [f42ccd0] FINISH, SetVdsStatusVDSCommand, log id: 5eba2505 >2013-03-08 14:11:05,944 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-6) [f42ccd0] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 14:11:05,965 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-6) [36f76f8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:11:05,968 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-6) [14e3210e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:11:05,969 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-6) [14e3210e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 14:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [57d9469d] Autorecovering 1 hosts >2013-03-08 14:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [57d9469d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [256baf81] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [256baf81] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [256baf81] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 31ffde73 >2013-03-08 14:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [256baf81] FINISH, SetVdsStatusVDSCommand, log id: 31ffde73 >2013-03-08 14:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-47) [256baf81] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2c547eb7 >2013-03-08 14:15:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [256baf81] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 482efd57 >2013-03-08 14:15:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [256baf81] FINISH, GetHardwareInfoVDSCommand, log id: 482efd57 >2013-03-08 14:15:00,087 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [4bf9f4ec] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:15:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [532c46c9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:15:00,091 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-47) [532c46c9] FINISH, ActivateVdsVDSCommand, log id: 2c547eb7 >2013-03-08 14:15:00,111 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [532c46c9] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:15:00,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [532c46c9] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 14:15:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [532c46c9] Autorecovering 1 storage domains >2013-03-08 14:15:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [532c46c9] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 14:15:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [757b1f7d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:15:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [757b1f7d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:15 PM >2013-03-08 14:15:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [757b1f7d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:15 PM >2013-03-08 14:15:00,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [13c58c68] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 39a32920 >2013-03-08 14:15:00,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [13c58c68] FINISH, GetHardwareInfoVDSCommand, log id: 39a32920 >2013-03-08 14:15:00,214 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-90) [44ea8c7c] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 14:15:00,218 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-90) [b539708] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 14:15:00,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4c59b7aa >2013-03-08 14:16:04,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 14:16:04,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:16:04,204 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 14:16:04,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 4c59b7aa >2013-03-08 14:16:04,208 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [249d90b3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:16:04,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [249d90b3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 309a3d19 >2013-03-08 14:16:04,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [249d90b3] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 14:16:04,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [249d90b3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 309a3d19 >2013-03-08 14:16:04,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [249d90b3] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 14:16:04,224 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [b539708] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 14:16:04,239 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-90) [2faaf8ed] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:16:04,240 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [2faaf8ed] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 31fd7081 >2013-03-08 14:16:04,241 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [2faaf8ed] FINISH, SetVdsStatusVDSCommand, log id: 31fd7081 >2013-03-08 14:16:04,247 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [2faaf8ed] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 14:16:04,268 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-90) [18c93eff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:16:04,271 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-90) [58ad55b5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:16:04,272 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-90) [58ad55b5] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 14:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) [4989b114] Autorecovering 1 hosts >2013-03-08 14:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) [4989b114] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-34) [4df8f2d6] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-34) [4df8f2d6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [4df8f2d6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 46b33e29 >2013-03-08 14:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [4df8f2d6] FINISH, SetVdsStatusVDSCommand, log id: 46b33e29 >2013-03-08 14:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-34) [4df8f2d6] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 45593882 >2013-03-08 14:20:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [4df8f2d6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2adb0416 >2013-03-08 14:20:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [4df8f2d6] FINISH, GetHardwareInfoVDSCommand, log id: 2adb0416 >2013-03-08 14:20:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-34) [c1022de] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:20:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-34) [5c2126df] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:20:00,203 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-34) [5c2126df] FINISH, ActivateVdsVDSCommand, log id: 45593882 >2013-03-08 14:20:00,222 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-34) [5c2126df] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:20:00,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-34) [5c2126df] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 14:20:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) [5c2126df] Autorecovering 1 storage domains >2013-03-08 14:20:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) [5c2126df] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 14:20:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-34) [557537b9] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:20:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-34) [557537b9] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:20 PM >2013-03-08 14:20:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-34) [557537b9] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:20 PM >2013-03-08 14:20:01,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [6da2ca63] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 21b17549 >2013-03-08 14:20:01,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [6da2ca63] FINISH, GetHardwareInfoVDSCommand, log id: 21b17549 >2013-03-08 14:20:01,549 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-66) [2c48561b] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 14:20:01,552 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-66) [760d6be1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 14:20:01,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4084de42 >2013-03-08 14:21:04,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 14:21:04,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:21:04,566 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 14:21:04,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 4084de42 >2013-03-08 14:21:04,570 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [2b6c9f1d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:21:04,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [2b6c9f1d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 36f22aa7 >2013-03-08 14:21:04,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [2b6c9f1d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 14:21:04,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [2b6c9f1d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 36f22aa7 >2013-03-08 14:21:04,586 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [2b6c9f1d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 14:21:04,586 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [760d6be1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 14:21:04,596 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-66) [3e298f37] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:21:04,597 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [3e298f37] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6860c42 >2013-03-08 14:21:04,599 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [3e298f37] FINISH, SetVdsStatusVDSCommand, log id: 6860c42 >2013-03-08 14:21:04,605 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-66) [3e298f37] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 14:21:04,626 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-66) [5dca3c15] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:21:04,630 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-66) [48418d1a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:21:04,631 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-66) [48418d1a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 14:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [e271f32] Autorecovering 1 hosts >2013-03-08 14:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [e271f32] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [20cb1aba] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [20cb1aba] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [20cb1aba] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 384fde32 >2013-03-08 14:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [20cb1aba] FINISH, SetVdsStatusVDSCommand, log id: 384fde32 >2013-03-08 14:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [20cb1aba] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 68c17467 >2013-03-08 14:25:00,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [20cb1aba] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 39af86d7 >2013-03-08 14:25:00,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [20cb1aba] FINISH, GetHardwareInfoVDSCommand, log id: 39af86d7 >2013-03-08 14:25:00,213 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-45) [553bc14a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:25:00,215 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-45) [4d816f35] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:25:00,217 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [4d816f35] FINISH, ActivateVdsVDSCommand, log id: 68c17467 >2013-03-08 14:25:00,236 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [4d816f35] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:25:00,237 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-45) [4d816f35] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 14:25:00,238 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [4d816f35] Autorecovering 1 storage domains >2013-03-08 14:25:00,238 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [4d816f35] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 14:25:00,239 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [2476f90c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:25:00,239 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [2476f90c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:25 PM >2013-03-08 14:25:00,242 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [2476f90c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:25 PM >2013-03-08 14:25:01,819 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [6be148e9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7dcca897 >2013-03-08 14:25:01,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [6be148e9] FINISH, GetHardwareInfoVDSCommand, log id: 7dcca897 >2013-03-08 14:25:01,877 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-1) [1b642806] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 14:25:01,880 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-1) [52a3e731] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 14:25:01,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 17f91edc >2013-03-08 14:26:07,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 14:26:07,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:26:07,931 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 14:26:07,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 17f91edc >2013-03-08 14:26:07,935 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [159e3e42] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:26:07,949 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [159e3e42] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3f6597a1 >2013-03-08 14:26:07,949 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [159e3e42] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 14:26:07,949 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [159e3e42] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3f6597a1 >2013-03-08 14:26:07,951 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [159e3e42] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 14:26:07,951 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [52a3e731] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 14:26:07,962 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-1) [1942761b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:26:07,963 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [1942761b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2f2a4e47 >2013-03-08 14:26:07,965 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [1942761b] FINISH, SetVdsStatusVDSCommand, log id: 2f2a4e47 >2013-03-08 14:26:07,970 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-1) [1942761b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 14:26:08,001 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-1) [3724ba90] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:26:08,003 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-1) [61ca503a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:26:08,005 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-1) [61ca503a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 14:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [1d287d1c] Autorecovering 1 hosts >2013-03-08 14:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [1d287d1c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [19963dcc] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [19963dcc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [19963dcc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 14a9a43b >2013-03-08 14:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [19963dcc] FINISH, SetVdsStatusVDSCommand, log id: 14a9a43b >2013-03-08 14:30:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [19963dcc] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2c8625f1 >2013-03-08 14:30:00,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [19963dcc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6543efea >2013-03-08 14:30:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [19963dcc] FINISH, GetHardwareInfoVDSCommand, log id: 6543efea >2013-03-08 14:30:00,258 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-42) [234bfb17] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:30:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-42) [4621f0bd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:30:00,261 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [4621f0bd] FINISH, ActivateVdsVDSCommand, log id: 2c8625f1 >2013-03-08 14:30:00,280 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [4621f0bd] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:30:00,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-42) [4621f0bd] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 14:30:00,282 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [4621f0bd] Autorecovering 1 storage domains >2013-03-08 14:30:00,282 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [4621f0bd] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 14:30:00,283 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [4c5485ab] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:30:00,283 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [4c5485ab] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:30 PM >2013-03-08 14:30:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [4c5485ab] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:30 PM >2013-03-08 14:30:02,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [76c1c41c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4af05ce2 >2013-03-08 14:30:02,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [76c1c41c] FINISH, GetHardwareInfoVDSCommand, log id: 4af05ce2 >2013-03-08 14:30:02,274 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-20) [3ad84686] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 14:30:02,278 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-20) [5b922646] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 14:30:02,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4b1cb3d1 >2013-03-08 14:30:33,744 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-63) [7819a10c] Start refreshing all users data >2013-03-08 14:30:34,692 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) [3ca1cd17] Attempting to update VMs/Templates Ovf. >2013-03-08 14:30:34,693 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) [3ca1cd17] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 14:30:34,693 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) [3ca1cd17] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 14:30:34,693 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) [3ca1cd17] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 14:30:34,694 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) [3ca1cd17] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 14:30:34,694 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) [3ca1cd17] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 14:30:34,695 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) [3ca1cd17] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 14:31:07,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 14:31:07,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:31:07,290 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 14:31:07,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 4b1cb3d1 >2013-03-08 14:31:07,294 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [6b1ea85d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:31:07,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [6b1ea85d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 25c4b92e >2013-03-08 14:31:07,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [6b1ea85d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 14:31:07,308 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [6b1ea85d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 25c4b92e >2013-03-08 14:31:07,310 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [6b1ea85d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 14:31:07,310 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [5b922646] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 14:31:07,320 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-20) [252fc410] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:31:07,321 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [252fc410] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 57eb815f >2013-03-08 14:31:07,322 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [252fc410] FINISH, SetVdsStatusVDSCommand, log id: 57eb815f >2013-03-08 14:31:07,328 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-20) [252fc410] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 14:31:07,353 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-20) [31ff30a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:31:07,356 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-20) [79bd4981] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:31:07,357 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-20) [79bd4981] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 14:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [5a01051a] Autorecovering 1 hosts >2013-03-08 14:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [5a01051a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [2481f8d8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [2481f8d8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [2481f8d8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 956a7ac >2013-03-08 14:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [2481f8d8] FINISH, SetVdsStatusVDSCommand, log id: 956a7ac >2013-03-08 14:35:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-35) [2481f8d8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2bcc949 >2013-03-08 14:35:00,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [2481f8d8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1f61e8c1 >2013-03-08 14:35:00,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [2481f8d8] FINISH, GetHardwareInfoVDSCommand, log id: 1f61e8c1 >2013-03-08 14:35:00,223 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-35) [60574607] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:35:00,225 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-35) [13e310fa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:35:00,227 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-35) [13e310fa] FINISH, ActivateVdsVDSCommand, log id: 2bcc949 >2013-03-08 14:35:00,246 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [13e310fa] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:35:00,247 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-35) [13e310fa] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 14:35:00,248 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [13e310fa] Autorecovering 1 storage domains >2013-03-08 14:35:00,248 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [13e310fa] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 14:35:00,249 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [2b2ebc80] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:35:00,249 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [2b2ebc80] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:35 PM >2013-03-08 14:35:00,252 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [2b2ebc80] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:35 PM >2013-03-08 14:35:01,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [73c0a37b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 400ab046 >2013-03-08 14:35:01,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [73c0a37b] FINISH, GetHardwareInfoVDSCommand, log id: 400ab046 >2013-03-08 14:35:01,596 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-15) [23a174f2] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 14:35:01,600 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-15) [606808fb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 14:35:01,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 83138c7 >2013-03-08 14:36:06,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 14:36:06,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:36:06,898 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 14:36:06,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 83138c7 >2013-03-08 14:36:06,901 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [63bfc5e0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:36:06,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [63bfc5e0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 90fcd71 >2013-03-08 14:36:06,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [63bfc5e0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 14:36:06,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [63bfc5e0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 90fcd71 >2013-03-08 14:36:06,917 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [63bfc5e0] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 14:36:06,917 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [606808fb] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 14:36:06,927 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-15) [3cf0e19] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:36:06,929 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [3cf0e19] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7fce0377 >2013-03-08 14:36:06,930 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [3cf0e19] FINISH, SetVdsStatusVDSCommand, log id: 7fce0377 >2013-03-08 14:36:06,936 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-15) [3cf0e19] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 14:36:06,957 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-15) [6158f99c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:36:06,959 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-15) [6ca6376] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:36:06,960 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-15) [6ca6376] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 14:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [27c46bf3] Autorecovering 1 hosts >2013-03-08 14:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [27c46bf3] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [491f52e6] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [491f52e6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [491f52e6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 79935706 >2013-03-08 14:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [491f52e6] FINISH, SetVdsStatusVDSCommand, log id: 79935706 >2013-03-08 14:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [491f52e6] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 61c376b1 >2013-03-08 14:40:00,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [491f52e6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 72066f2f >2013-03-08 14:40:00,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [491f52e6] FINISH, GetHardwareInfoVDSCommand, log id: 72066f2f >2013-03-08 14:40:00,623 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-16) [19909239] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:40:00,626 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-16) [36b275ed] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:40:00,627 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [36b275ed] FINISH, ActivateVdsVDSCommand, log id: 61c376b1 >2013-03-08 14:40:00,646 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [36b275ed] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:40:00,647 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-16) [36b275ed] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 14:40:00,648 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [36b275ed] Autorecovering 1 storage domains >2013-03-08 14:40:00,648 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [36b275ed] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 14:40:00,649 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [7161c0df] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:40:00,649 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [7161c0df] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:40 PM >2013-03-08 14:40:00,652 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [7161c0df] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:40 PM >2013-03-08 14:40:01,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [6f75aaff] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7217d7d >2013-03-08 14:40:01,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [6f75aaff] FINISH, GetHardwareInfoVDSCommand, log id: 7217d7d >2013-03-08 14:40:01,202 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-27) [5b6101f4] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 14:40:01,206 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-27) [3a0f7475] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 14:40:01,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4fade787 >2013-03-08 14:41:05,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 14:41:05,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:41:05,202 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 14:41:05,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 4fade787 >2013-03-08 14:41:05,205 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [2fd16087] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:41:05,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [2fd16087] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7ff5bce >2013-03-08 14:41:05,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [2fd16087] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 14:41:05,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [2fd16087] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7ff5bce >2013-03-08 14:41:05,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [2fd16087] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 14:41:05,222 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [3a0f7475] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 14:41:05,231 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-27) [6be349d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:41:05,233 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [6be349d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7aff5799 >2013-03-08 14:41:05,234 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [6be349d] FINISH, SetVdsStatusVDSCommand, log id: 7aff5799 >2013-03-08 14:41:05,239 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-27) [6be349d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 14:41:05,261 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-27) [9e2ca65] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:41:05,263 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-27) [7e7f6470] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:41:05,265 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-27) [7e7f6470] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 14:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [652c6408] Autorecovering 1 hosts >2013-03-08 14:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [652c6408] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [4e7ce2dc] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [4e7ce2dc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [4e7ce2dc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5007e87 >2013-03-08 14:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [4e7ce2dc] FINISH, SetVdsStatusVDSCommand, log id: 5007e87 >2013-03-08 14:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [4e7ce2dc] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 28135f9f >2013-03-08 14:45:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [4e7ce2dc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 36f6ea26 >2013-03-08 14:45:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [4e7ce2dc] FINISH, GetHardwareInfoVDSCommand, log id: 36f6ea26 >2013-03-08 14:45:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-69) [4566ace] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:45:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-69) [382c1150] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:45:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [382c1150] FINISH, ActivateVdsVDSCommand, log id: 28135f9f >2013-03-08 14:45:00,111 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [382c1150] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:45:00,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-69) [382c1150] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 14:45:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [382c1150] Autorecovering 1 storage domains >2013-03-08 14:45:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [382c1150] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 14:45:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [64e0fbe6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:45:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [64e0fbe6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:45 PM >2013-03-08 14:45:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [64e0fbe6] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:45 PM >2013-03-08 14:45:02,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [520061e8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: babd777 >2013-03-08 14:45:02,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [520061e8] FINISH, GetHardwareInfoVDSCommand, log id: babd777 >2013-03-08 14:45:02,510 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-75) [39e14355] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 14:45:02,514 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-75) [40cea279] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 14:45:02,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4115d1f0 >2013-03-08 14:46:07,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 14:46:07,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:46:07,542 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 14:46:07,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 4115d1f0 >2013-03-08 14:46:07,546 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [451900bf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:46:07,561 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [451900bf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4269f14c >2013-03-08 14:46:07,561 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [451900bf] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 14:46:07,561 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [451900bf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4269f14c >2013-03-08 14:46:07,563 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [451900bf] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 14:46:07,564 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [40cea279] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 14:46:07,573 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-75) [6f432ebc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:46:07,574 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [6f432ebc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 472b9dcb >2013-03-08 14:46:07,576 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [6f432ebc] FINISH, SetVdsStatusVDSCommand, log id: 472b9dcb >2013-03-08 14:46:07,581 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-75) [6f432ebc] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 14:46:07,602 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [32dcfc40] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:46:07,604 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [6facedac] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:46:07,606 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-75) [6facedac] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 14:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [30d2aa24] Autorecovering 1 hosts >2013-03-08 14:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [30d2aa24] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [1156e1b0] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [1156e1b0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [1156e1b0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2c4a0bdc >2013-03-08 14:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [1156e1b0] FINISH, SetVdsStatusVDSCommand, log id: 2c4a0bdc >2013-03-08 14:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-50) [1156e1b0] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5bb751f0 >2013-03-08 14:50:00,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [1156e1b0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 726cb5c1 >2013-03-08 14:50:00,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [1156e1b0] FINISH, GetHardwareInfoVDSCommand, log id: 726cb5c1 >2013-03-08 14:50:00,099 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-50) [6fe5ad88] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:50:00,102 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-50) [2a28f7b8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:50:00,103 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-50) [2a28f7b8] FINISH, ActivateVdsVDSCommand, log id: 5bb751f0 >2013-03-08 14:50:00,123 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [2a28f7b8] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:50:00,124 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-50) [2a28f7b8] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 14:50:00,125 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [2a28f7b8] Autorecovering 1 storage domains >2013-03-08 14:50:00,125 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [2a28f7b8] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 14:50:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [624a23cd] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:50:00,127 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [624a23cd] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:50 PM >2013-03-08 14:50:00,129 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [624a23cd] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:50 PM >2013-03-08 14:50:01,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [598f926d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 142257e9 >2013-03-08 14:50:01,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [598f926d] FINISH, GetHardwareInfoVDSCommand, log id: 142257e9 >2013-03-08 14:50:01,848 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-87) [53acdcc1] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 14:50:01,851 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-87) [18c61e49] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 14:50:01,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 30cc97cf >2013-03-08 14:51:06,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 14:51:06,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:51:06,883 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 14:51:06,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 30cc97cf >2013-03-08 14:51:06,887 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [7b7123ea] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:51:06,901 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [7b7123ea] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d2401a2 >2013-03-08 14:51:06,901 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [7b7123ea] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 14:51:06,901 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [7b7123ea] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d2401a2 >2013-03-08 14:51:06,906 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [18c61e49] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 14:51:06,916 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-87) [548116e4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:51:06,917 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [548116e4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 65e5bf2e >2013-03-08 14:51:06,918 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [548116e4] FINISH, SetVdsStatusVDSCommand, log id: 65e5bf2e >2013-03-08 14:51:06,924 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [548116e4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 14:51:06,944 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [75492811] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:51:06,946 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [5a31881] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:51:06,947 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-87) [5a31881] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 14:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [25f71f5a] Autorecovering 1 hosts >2013-03-08 14:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [25f71f5a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [52bd52dd] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [52bd52dd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [52bd52dd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 63a7a6ec >2013-03-08 14:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [52bd52dd] FINISH, SetVdsStatusVDSCommand, log id: 63a7a6ec >2013-03-08 14:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-92) [52bd52dd] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4c21865e >2013-03-08 14:55:00,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [52bd52dd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1b4495f3 >2013-03-08 14:55:00,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [52bd52dd] FINISH, GetHardwareInfoVDSCommand, log id: 1b4495f3 >2013-03-08 14:55:00,094 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-92) [5eb9783a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:55:00,097 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-92) [44e3c0f7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:55:00,098 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-92) [44e3c0f7] FINISH, ActivateVdsVDSCommand, log id: 4c21865e >2013-03-08 14:55:00,117 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [44e3c0f7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 14:55:00,118 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-92) [44e3c0f7] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 14:55:00,119 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [44e3c0f7] Autorecovering 1 storage domains >2013-03-08 14:55:00,119 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [44e3c0f7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 14:55:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [10761322] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:55:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [10761322] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 2:55 PM >2013-03-08 14:55:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [10761322] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 2:55 PM >2013-03-08 14:55:01,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [5505c21c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 23981b6f >2013-03-08 14:55:01,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [5505c21c] FINISH, GetHardwareInfoVDSCommand, log id: 23981b6f >2013-03-08 14:55:01,193 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-49) [479b5ff7] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 14:55:01,197 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-49) [4dba17df] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 14:55:01,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 492d37d7 >2013-03-08 14:56:07,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 14:56:07,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 14:56:07,233 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 14:56:07,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 492d37d7 >2013-03-08 14:56:07,236 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [f0ff8df] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 14:56:07,252 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [f0ff8df] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 49ad1913 >2013-03-08 14:56:07,252 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [f0ff8df] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 14:56:07,252 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [f0ff8df] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 49ad1913 >2013-03-08 14:56:07,254 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [f0ff8df] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 14:56:07,254 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [4dba17df] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 14:56:07,264 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-49) [53a2c370] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:56:07,265 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [53a2c370] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 150c0f5 >2013-03-08 14:56:07,267 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [53a2c370] FINISH, SetVdsStatusVDSCommand, log id: 150c0f5 >2013-03-08 14:56:07,272 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-49) [53a2c370] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 14:56:07,293 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-49) [3427e877] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:56:07,296 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-49) [71ee8c1f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 14:56:07,297 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) [71ee8c1f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 15:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [7dc0a8a] Autorecovering 1 hosts >2013-03-08 15:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [7dc0a8a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [1bd9fc91] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [1bd9fc91] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [1bd9fc91] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6134430e >2013-03-08 15:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [1bd9fc91] FINISH, SetVdsStatusVDSCommand, log id: 6134430e >2013-03-08 15:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [1bd9fc91] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3f99cda9 >2013-03-08 15:00:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [1bd9fc91] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: a97f12d >2013-03-08 15:00:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [1bd9fc91] FINISH, GetHardwareInfoVDSCommand, log id: a97f12d >2013-03-08 15:00:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [7775e384] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:00:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [718eb118] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:00:00,093 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [718eb118] FINISH, ActivateVdsVDSCommand, log id: 3f99cda9 >2013-03-08 15:00:00,112 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [718eb118] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:00:00,113 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [718eb118] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 15:00:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [718eb118] Autorecovering 1 storage domains >2013-03-08 15:00:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [718eb118] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 15:00:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [31bb714e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:00:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [31bb714e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:00 PM >2013-03-08 15:00:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [31bb714e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:00 PM >2013-03-08 15:00:01,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-62) [1cbb8e9a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1c718ce4 >2013-03-08 15:00:01,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-62) [1cbb8e9a] FINISH, GetHardwareInfoVDSCommand, log id: 1c718ce4 >2013-03-08 15:00:01,547 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-62) [33d19454] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 15:00:01,550 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-62) [a8384eb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 15:00:01,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 54710e8a >2013-03-08 15:01:06,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 15:01:06,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:01:06,616 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 15:01:06,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 54710e8a >2013-03-08 15:01:06,620 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [4a855e73] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:01:06,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4a855e73] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4cc9efcf >2013-03-08 15:01:06,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [4a855e73] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 15:01:06,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4a855e73] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4cc9efcf >2013-03-08 15:01:06,642 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [4a855e73] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 15:01:06,643 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [a8384eb] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 15:01:06,653 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-62) [cf6cb83] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:01:06,654 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-62) [cf6cb83] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3af8a6f4 >2013-03-08 15:01:06,655 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-62) [cf6cb83] FINISH, SetVdsStatusVDSCommand, log id: 3af8a6f4 >2013-03-08 15:01:06,662 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-62) [cf6cb83] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 15:01:06,683 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-62) [428ba913] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:01:06,685 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-62) [7685c42e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:01:06,687 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-62) [7685c42e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 15:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [2cc208aa] Autorecovering 1 hosts >2013-03-08 15:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [2cc208aa] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [670567e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [670567e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [670567e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 483f6d79 >2013-03-08 15:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [670567e] FINISH, SetVdsStatusVDSCommand, log id: 483f6d79 >2013-03-08 15:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [670567e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 254663a5 >2013-03-08 15:05:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [670567e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 468ac8ce >2013-03-08 15:05:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [670567e] FINISH, GetHardwareInfoVDSCommand, log id: 468ac8ce >2013-03-08 15:05:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-14) [21e7aca2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:05:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-14) [63074ef9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:05:00,094 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [63074ef9] FINISH, ActivateVdsVDSCommand, log id: 254663a5 >2013-03-08 15:05:00,113 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [63074ef9] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:05:00,114 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-14) [63074ef9] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 15:05:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [63074ef9] Autorecovering 1 storage domains >2013-03-08 15:05:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [63074ef9] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 15:05:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [39b93541] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:05:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [39b93541] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:05 PM >2013-03-08 15:05:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [39b93541] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:05 PM >2013-03-08 15:05:00,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [fb36739] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 52c2a82e >2013-03-08 15:05:00,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [fb36739] FINISH, GetHardwareInfoVDSCommand, log id: 52c2a82e >2013-03-08 15:05:00,930 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-25) [2e4712ad] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 15:05:00,933 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-25) [54698061] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 15:05:00,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 73d9a6fe >2013-03-08 15:06:04,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 15:06:04,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:06:04,938 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 15:06:04,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 73d9a6fe >2013-03-08 15:06:04,942 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [1601c936] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:06:04,956 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [1601c936] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 67a54e37 >2013-03-08 15:06:04,956 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [1601c936] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 15:06:04,956 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [1601c936] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 67a54e37 >2013-03-08 15:06:04,958 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [1601c936] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 15:06:04,959 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [54698061] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 15:06:04,968 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-25) [9834b4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:06:04,969 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [9834b4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 68626651 >2013-03-08 15:06:04,970 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [9834b4] FINISH, SetVdsStatusVDSCommand, log id: 68626651 >2013-03-08 15:06:04,976 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-25) [9834b4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 15:06:04,997 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-25) [39948ddf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:06:05,000 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-25) [79c154e4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:06:05,001 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-25) [79c154e4] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 15:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [12230346] Autorecovering 1 hosts >2013-03-08 15:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [12230346] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-95) [6a29ca44] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-95) [6a29ca44] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [6a29ca44] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4fa3b6db >2013-03-08 15:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [6a29ca44] FINISH, SetVdsStatusVDSCommand, log id: 4fa3b6db >2013-03-08 15:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-95) [6a29ca44] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2b3f240b >2013-03-08 15:10:00,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [6a29ca44] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7eea9938 >2013-03-08 15:10:00,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [6a29ca44] FINISH, GetHardwareInfoVDSCommand, log id: 7eea9938 >2013-03-08 15:10:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-95) [223ffd3c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:10:00,094 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-95) [4e578777] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:10:00,096 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-95) [4e578777] FINISH, ActivateVdsVDSCommand, log id: 2b3f240b >2013-03-08 15:10:00,115 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-95) [4e578777] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:10:00,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-95) [4e578777] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 15:10:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [4e578777] Autorecovering 1 storage domains >2013-03-08 15:10:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [4e578777] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 15:10:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-95) [50e78925] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:10:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-95) [50e78925] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:10 PM >2013-03-08 15:10:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-95) [50e78925] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:10 PM >2013-03-08 15:10:02,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [207c5a5b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4097258a >2013-03-08 15:10:02,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [207c5a5b] FINISH, GetHardwareInfoVDSCommand, log id: 4097258a >2013-03-08 15:10:02,251 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-9) [22e172f4] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 15:10:02,254 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-9) [475bde3f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 15:10:02,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 42f8fdb9 >2013-03-08 15:11:08,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 15:11:08,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:11:08,242 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 15:11:08,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 42f8fdb9 >2013-03-08 15:11:08,246 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [6a5d4d69] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:11:08,259 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6a5d4d69] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 33bc9e64 >2013-03-08 15:11:08,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [6a5d4d69] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 15:11:08,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6a5d4d69] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 33bc9e64 >2013-03-08 15:11:08,262 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [6a5d4d69] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 15:11:08,262 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [475bde3f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 15:11:08,271 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-9) [7cd31711] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:11:08,273 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [7cd31711] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3e120baa >2013-03-08 15:11:08,274 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [7cd31711] FINISH, SetVdsStatusVDSCommand, log id: 3e120baa >2013-03-08 15:11:08,280 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-9) [7cd31711] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 15:11:08,302 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-9) [386102f8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:11:08,305 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-9) [6100460d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:11:08,306 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-9) [6100460d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 15:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [b60a8e3] Autorecovering 1 hosts >2013-03-08 15:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [b60a8e3] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [1435ae07] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [1435ae07] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [1435ae07] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3418738e >2013-03-08 15:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [1435ae07] FINISH, SetVdsStatusVDSCommand, log id: 3418738e >2013-03-08 15:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [1435ae07] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1e9e087 >2013-03-08 15:15:00,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [1435ae07] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6093bd7d >2013-03-08 15:15:00,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [1435ae07] FINISH, GetHardwareInfoVDSCommand, log id: 6093bd7d >2013-03-08 15:15:00,185 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-79) [3d564ef9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:15:00,187 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-79) [16c9ea1d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:15:00,188 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [16c9ea1d] FINISH, ActivateVdsVDSCommand, log id: 1e9e087 >2013-03-08 15:15:00,208 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [16c9ea1d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:15:00,209 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-79) [16c9ea1d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 15:15:00,210 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [16c9ea1d] Autorecovering 1 storage domains >2013-03-08 15:15:00,210 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [16c9ea1d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 15:15:00,210 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [2323f6cd] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:15:00,211 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [2323f6cd] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:15 PM >2013-03-08 15:15:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [2323f6cd] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:15 PM >2013-03-08 15:15:02,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-23) [4edfb6ce] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 774f48ff >2013-03-08 15:15:02,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-23) [4edfb6ce] FINISH, GetHardwareInfoVDSCommand, log id: 774f48ff >2013-03-08 15:15:02,547 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-23) [333161a3] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 15:15:02,550 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-23) [514b8708] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 15:15:02,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4e04f744 >2013-03-08 15:16:05,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 15:16:05,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:16:05,544 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 15:16:05,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 4e04f744 >2013-03-08 15:16:05,548 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [31f5b852] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:16:05,561 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [31f5b852] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 402f13c5 >2013-03-08 15:16:05,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [31f5b852] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 15:16:05,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [31f5b852] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 402f13c5 >2013-03-08 15:16:05,564 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [31f5b852] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 15:16:05,564 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [514b8708] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 15:16:05,574 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-23) [700e5a19] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:16:05,576 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-23) [700e5a19] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 62de7ff0 >2013-03-08 15:16:05,577 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-23) [700e5a19] FINISH, SetVdsStatusVDSCommand, log id: 62de7ff0 >2013-03-08 15:16:05,582 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-23) [700e5a19] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 15:16:05,620 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-23) [31b9fe07] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:16:05,623 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-23) [1b1b0943] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:16:05,624 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-23) [1b1b0943] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 15:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [4a6c8eaf] Autorecovering 1 hosts >2013-03-08 15:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [4a6c8eaf] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [3b5c136d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [3b5c136d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [3b5c136d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7d5ed1f5 >2013-03-08 15:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [3b5c136d] FINISH, SetVdsStatusVDSCommand, log id: 7d5ed1f5 >2013-03-08 15:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-32) [3b5c136d] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 27017fda >2013-03-08 15:20:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [3b5c136d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4e3edd45 >2013-03-08 15:20:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [3b5c136d] FINISH, GetHardwareInfoVDSCommand, log id: 4e3edd45 >2013-03-08 15:20:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-32) [342751e8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:20:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-32) [53e8c1b6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:20:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-32) [53e8c1b6] FINISH, ActivateVdsVDSCommand, log id: 27017fda >2013-03-08 15:20:00,222 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [53e8c1b6] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:20:00,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-32) [53e8c1b6] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 15:20:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [53e8c1b6] Autorecovering 1 storage domains >2013-03-08 15:20:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [53e8c1b6] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 15:20:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [248f5a32] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:20:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [248f5a32] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:20 PM >2013-03-08 15:20:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [248f5a32] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:20 PM >2013-03-08 15:20:02,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [688afdbc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 15a3ea3f >2013-03-08 15:20:02,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [688afdbc] FINISH, GetHardwareInfoVDSCommand, log id: 15a3ea3f >2013-03-08 15:20:02,874 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-46) [4ecf75db] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 15:20:02,878 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-46) [46faf5f8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 15:20:02,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 452028c5 >2013-03-08 15:21:08,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 15:21:08,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:21:08,874 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 15:21:08,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 452028c5 >2013-03-08 15:21:08,878 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [5d247015] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:21:08,891 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5d247015] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6d973791 >2013-03-08 15:21:08,891 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [5d247015] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 15:21:08,891 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5d247015] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6d973791 >2013-03-08 15:21:08,893 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [5d247015] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 15:21:08,894 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [46faf5f8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 15:21:08,903 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-46) [e0a6ad0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:21:08,904 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [e0a6ad0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2f3ce26f >2013-03-08 15:21:08,906 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [e0a6ad0] FINISH, SetVdsStatusVDSCommand, log id: 2f3ce26f >2013-03-08 15:21:08,911 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-46) [e0a6ad0] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 15:21:08,932 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-46) [24dfdb7b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:21:08,934 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-46) [58449771] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:21:08,936 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-46) [58449771] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 15:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [20f4a0a1] Autorecovering 1 hosts >2013-03-08 15:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [20f4a0a1] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [3fe46ad8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [3fe46ad8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [3fe46ad8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5a1d17a >2013-03-08 15:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [3fe46ad8] FINISH, SetVdsStatusVDSCommand, log id: 5a1d17a >2013-03-08 15:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-13) [3fe46ad8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3900a14 >2013-03-08 15:25:00,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [3fe46ad8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1ef45641 >2013-03-08 15:25:00,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [3fe46ad8] FINISH, GetHardwareInfoVDSCommand, log id: 1ef45641 >2013-03-08 15:25:00,208 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-13) [785daaf7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:25:00,211 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-13) [4e427cfd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:25:00,212 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-13) [4e427cfd] FINISH, ActivateVdsVDSCommand, log id: 3900a14 >2013-03-08 15:25:00,231 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [4e427cfd] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:25:00,232 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-13) [4e427cfd] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 15:25:00,233 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [4e427cfd] Autorecovering 1 storage domains >2013-03-08 15:25:00,234 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [4e427cfd] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 15:25:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [765ac3f2] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:25:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [765ac3f2] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:25 PM >2013-03-08 15:25:00,237 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [765ac3f2] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:25 PM >2013-03-08 15:25:00,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [58ad55b5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2d02bff3 >2013-03-08 15:25:00,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [58ad55b5] FINISH, GetHardwareInfoVDSCommand, log id: 2d02bff3 >2013-03-08 15:25:00,309 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-90) [42d3d939] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 15:25:00,313 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-90) [47c368e3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 15:25:00,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4177fdc0 >2013-03-08 15:26:04,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 15:26:04,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:26:04,380 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 15:26:04,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 4177fdc0 >2013-03-08 15:26:04,383 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [11a234ae] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:26:04,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [11a234ae] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 237f015d >2013-03-08 15:26:04,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [11a234ae] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 15:26:04,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [11a234ae] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 237f015d >2013-03-08 15:26:04,399 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [11a234ae] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 15:26:04,400 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [47c368e3] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 15:26:04,409 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-90) [22cabf3e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:26:04,410 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [22cabf3e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 60368e6d >2013-03-08 15:26:04,412 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [22cabf3e] FINISH, SetVdsStatusVDSCommand, log id: 60368e6d >2013-03-08 15:26:04,418 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [22cabf3e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 15:26:04,445 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-90) [3b189d70] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:26:04,447 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-90) [2fa45b6a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:26:04,448 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-90) [2fa45b6a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 15:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [6ae64941] Autorecovering 1 hosts >2013-03-08 15:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [6ae64941] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-28) [42849800] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-28) [42849800] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [42849800] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 695ff606 >2013-03-08 15:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [42849800] FINISH, SetVdsStatusVDSCommand, log id: 695ff606 >2013-03-08 15:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-28) [42849800] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2417b1a3 >2013-03-08 15:30:00,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [42849800] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6b1f1e2b >2013-03-08 15:30:00,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [42849800] FINISH, GetHardwareInfoVDSCommand, log id: 6b1f1e2b >2013-03-08 15:30:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-28) [7dd2e4c9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:30:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-28) [473fc099] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:30:00,205 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-28) [473fc099] FINISH, ActivateVdsVDSCommand, log id: 2417b1a3 >2013-03-08 15:30:00,225 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-28) [473fc099] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:30:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [473fc099] Autorecovering 1 storage domains >2013-03-08 15:30:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [473fc099] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 15:30:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-28) [194ad8e9] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:30:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-28) [194ad8e9] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:30 PM >2013-03-08 15:30:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-28) [194ad8e9] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:30 PM >2013-03-08 15:30:01,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [48418d1a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 472df09 >2013-03-08 15:30:01,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [48418d1a] FINISH, GetHardwareInfoVDSCommand, log id: 472df09 >2013-03-08 15:30:01,695 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-66) [53dbec05] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 15:30:01,699 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-66) [57fbc955] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 15:30:01,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 52335673 >2013-03-08 15:30:33,746 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-21) [54a87db] Start refreshing all users data >2013-03-08 15:30:34,695 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-16) [7161c0df] Attempting to update VMs/Templates Ovf. >2013-03-08 15:30:34,696 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-16) [7161c0df] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 15:30:34,697 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-16) [7161c0df] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 15:30:34,697 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-16) [7161c0df] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 15:30:34,698 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-16) [7161c0df] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 15:30:34,698 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-16) [7161c0df] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 15:30:34,698 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-16) [7161c0df] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 15:31:06,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 15:31:06,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:31:06,747 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 15:31:06,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 52335673 >2013-03-08 15:31:06,751 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [268b36ee] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:31:06,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [268b36ee] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 749b754f >2013-03-08 15:31:06,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [268b36ee] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 15:31:06,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [268b36ee] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 749b754f >2013-03-08 15:31:06,768 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [268b36ee] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 15:31:06,768 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [57fbc955] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 15:31:06,778 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-66) [1858d4a6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:31:06,779 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [1858d4a6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 36ebc797 >2013-03-08 15:31:06,780 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [1858d4a6] FINISH, SetVdsStatusVDSCommand, log id: 36ebc797 >2013-03-08 15:31:06,786 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-66) [1858d4a6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 15:31:06,821 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-66) [1b843bb4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:31:06,824 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-66) [4378fd6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:31:06,825 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-66) [4378fd6] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 15:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [8e7cbd9] Autorecovering 1 hosts >2013-03-08 15:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [8e7cbd9] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-38) [64296a4] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-38) [64296a4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [64296a4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 598af6a4 >2013-03-08 15:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [64296a4] FINISH, SetVdsStatusVDSCommand, log id: 598af6a4 >2013-03-08 15:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-38) [64296a4] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4d49efd0 >2013-03-08 15:35:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [64296a4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7e41bb3d >2013-03-08 15:35:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [64296a4] FINISH, GetHardwareInfoVDSCommand, log id: 7e41bb3d >2013-03-08 15:35:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-38) [2fb3a0cb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:35:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-38) [4f30b5c0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:35:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-38) [4f30b5c0] FINISH, ActivateVdsVDSCommand, log id: 4d49efd0 >2013-03-08 15:35:00,221 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-38) [4f30b5c0] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:35:00,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-38) [4f30b5c0] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 15:35:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [4f30b5c0] Autorecovering 1 storage domains >2013-03-08 15:35:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [4f30b5c0] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 15:35:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-38) [1c4e755b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:35:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-38) [1c4e755b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:35 PM >2013-03-08 15:35:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-38) [1c4e755b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:35 PM >2013-03-08 15:35:01,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [294d68be] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: fd82b9e >2013-03-08 15:35:01,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [294d68be] FINISH, GetHardwareInfoVDSCommand, log id: fd82b9e >2013-03-08 15:35:01,064 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-53) [7564bd40] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 15:35:01,067 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-53) [6482d79d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 15:35:01,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 13b44118 >2013-03-08 15:36:05,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 15:36:05,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:36:05,343 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 15:36:05,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 13b44118 >2013-03-08 15:36:05,347 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [21d68f1c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:36:05,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [21d68f1c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5760455c >2013-03-08 15:36:05,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [21d68f1c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 15:36:05,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [21d68f1c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5760455c >2013-03-08 15:36:05,363 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [21d68f1c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 15:36:05,363 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [6482d79d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 15:36:05,373 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-53) [378e7abd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:36:05,375 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [378e7abd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: d5948e0 >2013-03-08 15:36:05,376 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [378e7abd] FINISH, SetVdsStatusVDSCommand, log id: d5948e0 >2013-03-08 15:36:05,382 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-53) [378e7abd] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 15:36:05,402 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-53) [1e3bfa88] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:36:05,404 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-53) [c6191f9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:36:05,406 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-53) [c6191f9] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 15:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [12f9356a] Autorecovering 1 hosts >2013-03-08 15:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [12f9356a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [3b554857] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [3b554857] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [3b554857] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 10f84e8c >2013-03-08 15:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [3b554857] FINISH, SetVdsStatusVDSCommand, log id: 10f84e8c >2013-03-08 15:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-18) [3b554857] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 20d70af3 >2013-03-08 15:40:00,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [3b554857] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 64e44564 >2013-03-08 15:40:00,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [3b554857] FINISH, GetHardwareInfoVDSCommand, log id: 64e44564 >2013-03-08 15:40:00,185 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [3e38d626] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:40:00,187 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [7291088] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:40:00,188 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-18) [7291088] FINISH, ActivateVdsVDSCommand, log id: 20d70af3 >2013-03-08 15:40:00,207 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [7291088] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:40:00,208 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [7291088] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 15:40:00,210 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [7291088] Autorecovering 1 storage domains >2013-03-08 15:40:00,210 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [7291088] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 15:40:00,210 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [fff7e03] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:40:00,210 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [fff7e03] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:40 PM >2013-03-08 15:40:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [fff7e03] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:40 PM >2013-03-08 15:40:02,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [194ad8e9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4196884b >2013-03-08 15:40:02,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [194ad8e9] FINISH, GetHardwareInfoVDSCommand, log id: 4196884b >2013-03-08 15:40:02,648 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-28) [66135bb5] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 15:40:02,652 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-28) [7ad5d5f8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 15:40:02,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2a384d76 >2013-03-08 15:41:06,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 15:41:06,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:41:06,679 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 15:41:06,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2a384d76 >2013-03-08 15:41:06,683 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [7df90b28] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:41:06,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [7df90b28] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5b357cc4 >2013-03-08 15:41:06,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [7df90b28] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 15:41:06,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [7df90b28] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5b357cc4 >2013-03-08 15:41:06,699 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [7df90b28] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 15:41:06,699 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [7ad5d5f8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 15:41:06,709 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-28) [2ee7f0b8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:41:06,711 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [2ee7f0b8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 52149efe >2013-03-08 15:41:06,712 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [2ee7f0b8] FINISH, SetVdsStatusVDSCommand, log id: 52149efe >2013-03-08 15:41:06,717 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-28) [2ee7f0b8] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 15:41:06,739 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-28) [7d3f3986] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:41:06,741 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-28) [368aa39f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:41:06,743 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-28) [368aa39f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 15:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [651ec7ed] Autorecovering 1 hosts >2013-03-08 15:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [651ec7ed] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-99) [275b10fd] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-99) [275b10fd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [275b10fd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 33548628 >2013-03-08 15:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [275b10fd] FINISH, SetVdsStatusVDSCommand, log id: 33548628 >2013-03-08 15:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-99) [275b10fd] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 667c56da >2013-03-08 15:45:00,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [275b10fd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2403299f >2013-03-08 15:45:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [275b10fd] FINISH, GetHardwareInfoVDSCommand, log id: 2403299f >2013-03-08 15:45:00,257 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-99) [20823509] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:45:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-99) [61eeead4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:45:00,261 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-99) [61eeead4] FINISH, ActivateVdsVDSCommand, log id: 667c56da >2013-03-08 15:45:00,287 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-99) [61eeead4] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:45:00,287 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-99) [61eeead4] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 15:45:00,289 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [61eeead4] Autorecovering 1 storage domains >2013-03-08 15:45:00,289 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [61eeead4] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 15:45:00,289 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-99) [42313555] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:45:00,289 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-99) [42313555] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:45 PM >2013-03-08 15:45:00,292 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-99) [42313555] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:45 PM >2013-03-08 15:45:00,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [5461d040] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6f0e923a >2013-03-08 15:45:00,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [5461d040] FINISH, GetHardwareInfoVDSCommand, log id: 6f0e923a >2013-03-08 15:45:00,988 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-36) [6918c360] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 15:45:00,992 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-36) [44877b1c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 15:45:01,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5964d10d >2013-03-08 15:46:05,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 15:46:05,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:46:05,973 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 15:46:05,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 5964d10d >2013-03-08 15:46:05,977 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [6dc67ec1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:46:05,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6dc67ec1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: c640ad2 >2013-03-08 15:46:05,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [6dc67ec1] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 15:46:05,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6dc67ec1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: c640ad2 >2013-03-08 15:46:05,995 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [6dc67ec1] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 15:46:05,995 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [44877b1c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 15:46:06,005 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-36) [14cc597b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:46:06,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [14cc597b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3da2826e >2013-03-08 15:46:06,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [14cc597b] FINISH, SetVdsStatusVDSCommand, log id: 3da2826e >2013-03-08 15:46:06,014 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [14cc597b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 15:46:06,034 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [2a64acb1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:46:06,037 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [16379607] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:46:06,038 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-36) [16379607] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 15:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [2476f90c] Autorecovering 1 hosts >2013-03-08 15:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [2476f90c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [ac2fd23] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [ac2fd23] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [ac2fd23] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2ecbc9fb >2013-03-08 15:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [ac2fd23] FINISH, SetVdsStatusVDSCommand, log id: 2ecbc9fb >2013-03-08 15:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [ac2fd23] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4ab2a3ed >2013-03-08 15:50:00,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [ac2fd23] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 447a9a13 >2013-03-08 15:50:00,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [ac2fd23] FINISH, GetHardwareInfoVDSCommand, log id: 447a9a13 >2013-03-08 15:50:00,191 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-45) [60e6eb9c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:50:00,194 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-45) [2fea1746] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:50:00,195 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [2fea1746] FINISH, ActivateVdsVDSCommand, log id: 4ab2a3ed >2013-03-08 15:50:00,215 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [2fea1746] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:50:00,216 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-45) [2fea1746] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 15:50:00,217 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [2fea1746] Autorecovering 1 storage domains >2013-03-08 15:50:00,217 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [2fea1746] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 15:50:00,218 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [3f7ba047] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:50:00,218 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [3f7ba047] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:50 PM >2013-03-08 15:50:00,221 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [3f7ba047] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:50 PM >2013-03-08 15:50:00,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [75097004] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1af03298 >2013-03-08 15:50:00,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [75097004] FINISH, GetHardwareInfoVDSCommand, log id: 1af03298 >2013-03-08 15:50:00,293 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-4) [4c4d1790] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 15:50:00,296 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-4) [1b3dfe2e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 15:50:00,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 512b3c16 >2013-03-08 15:51:04,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 15:51:04,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:51:04,311 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 15:51:04,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 512b3c16 >2013-03-08 15:51:04,314 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [404a2582] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:51:04,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [404a2582] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4f7aeddb >2013-03-08 15:51:04,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [404a2582] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 15:51:04,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [404a2582] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4f7aeddb >2013-03-08 15:51:04,330 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [404a2582] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 15:51:04,331 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [1b3dfe2e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 15:51:04,340 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-4) [67aaca71] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:51:04,342 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [67aaca71] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 30116b5d >2013-03-08 15:51:04,343 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [67aaca71] FINISH, SetVdsStatusVDSCommand, log id: 30116b5d >2013-03-08 15:51:04,349 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-4) [67aaca71] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 15:51:04,370 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-4) [6aaa7312] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:51:04,372 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-4) [47e90346] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:51:04,374 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-4) [47e90346] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 15:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [e8ee453] Autorecovering 1 hosts >2013-03-08 15:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [e8ee453] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [4f97ca34] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [4f97ca34] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [4f97ca34] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 422caa99 >2013-03-08 15:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [4f97ca34] FINISH, SetVdsStatusVDSCommand, log id: 422caa99 >2013-03-08 15:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-64) [4f97ca34] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3d9ca8d8 >2013-03-08 15:55:00,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [4f97ca34] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 42fdddc >2013-03-08 15:55:00,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [4f97ca34] FINISH, GetHardwareInfoVDSCommand, log id: 42fdddc >2013-03-08 15:55:00,302 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-64) [5cdbfa82] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:55:00,304 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-64) [551e6d97] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:55:00,305 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-64) [551e6d97] FINISH, ActivateVdsVDSCommand, log id: 3d9ca8d8 >2013-03-08 15:55:00,324 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [551e6d97] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 15:55:00,325 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-64) [551e6d97] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 15:55:00,326 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [551e6d97] Autorecovering 1 storage domains >2013-03-08 15:55:00,327 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [551e6d97] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 15:55:00,327 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [1e82ea3b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:55:00,327 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [1e82ea3b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 3:55 PM >2013-03-08 15:55:00,330 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [1e82ea3b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 3:55 PM >2013-03-08 15:55:01,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [557537b9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 116aa24e >2013-03-08 15:55:01,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [557537b9] FINISH, GetHardwareInfoVDSCommand, log id: 116aa24e >2013-03-08 15:55:01,617 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-34) [4e268d28] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 15:55:01,620 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-34) [70918eb8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 15:55:01,630 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3c928557 >2013-03-08 15:56:06,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 15:56:06,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 15:56:06,668 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 15:56:06,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 3c928557 >2013-03-08 15:56:06,672 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [6452edb5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 15:56:06,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [6452edb5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2e311a3d >2013-03-08 15:56:06,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [6452edb5] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 15:56:06,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [6452edb5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2e311a3d >2013-03-08 15:56:06,691 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [70918eb8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 15:56:06,701 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-34) [21846b97] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:56:06,702 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [21846b97] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 587c4d89 >2013-03-08 15:56:06,703 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [21846b97] FINISH, SetVdsStatusVDSCommand, log id: 587c4d89 >2013-03-08 15:56:06,709 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-34) [21846b97] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 15:56:06,730 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-34) [28021f60] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:56:06,733 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-34) [626594b9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 15:56:06,734 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) [626594b9] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 16:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [353e723a] Autorecovering 1 hosts >2013-03-08 16:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [353e723a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [3291cfbd] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [3291cfbd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [3291cfbd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 24ef4524 >2013-03-08 16:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [3291cfbd] FINISH, SetVdsStatusVDSCommand, log id: 24ef4524 >2013-03-08 16:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [3291cfbd] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 39b5e55 >2013-03-08 16:00:00,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [3291cfbd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 38abaf94 >2013-03-08 16:00:00,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [3291cfbd] FINISH, GetHardwareInfoVDSCommand, log id: 38abaf94 >2013-03-08 16:00:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [7da428b4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:00:00,207 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [13f75266] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:00:00,209 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [13f75266] FINISH, ActivateVdsVDSCommand, log id: 39b5e55 >2013-03-08 16:00:00,228 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [13f75266] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:00:00,229 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [13f75266] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 16:00:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [13f75266] Autorecovering 1 storage domains >2013-03-08 16:00:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [13f75266] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 16:00:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [2aee83e2] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:00:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [2aee83e2] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:00 PM >2013-03-08 16:00:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [2aee83e2] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:00 PM >2013-03-08 16:00:00,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [2b2ebc80] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1000d8c5 >2013-03-08 16:00:00,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [2b2ebc80] FINISH, GetHardwareInfoVDSCommand, log id: 1000d8c5 >2013-03-08 16:00:00,978 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-35) [237925f1] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 16:00:00,982 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-35) [554e85c2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 16:00:00,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: f8a44cb >2013-03-08 16:01:05,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 16:01:05,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:01:05,996 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 16:01:05,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: f8a44cb >2013-03-08 16:01:05,999 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [7ca53c6b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:01:06,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [7ca53c6b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: ba08817 >2013-03-08 16:01:06,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [7ca53c6b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 16:01:06,013 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [7ca53c6b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: ba08817 >2013-03-08 16:01:06,015 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [7ca53c6b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 16:01:06,015 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [554e85c2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 16:01:06,026 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-35) [3c565abc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:01:06,027 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [3c565abc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2f67c0a9 >2013-03-08 16:01:06,028 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [3c565abc] FINISH, SetVdsStatusVDSCommand, log id: 2f67c0a9 >2013-03-08 16:01:06,035 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-35) [3c565abc] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 16:01:06,057 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-35) [60c23560] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:01:06,059 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-35) [20753663] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:01:06,061 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-35) [20753663] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 16:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [4c5485ab] Autorecovering 1 hosts >2013-03-08 16:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [4c5485ab] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [2a483c0c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [2a483c0c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [2a483c0c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 455a19e0 >2013-03-08 16:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [2a483c0c] FINISH, SetVdsStatusVDSCommand, log id: 455a19e0 >2013-03-08 16:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [2a483c0c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 14a8a6ec >2013-03-08 16:05:00,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [2a483c0c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3b4cd3c2 >2013-03-08 16:05:00,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [2a483c0c] FINISH, GetHardwareInfoVDSCommand, log id: 3b4cd3c2 >2013-03-08 16:05:00,185 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-42) [38a82c88] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:05:00,187 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-42) [37206ed6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:05:00,188 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [37206ed6] FINISH, ActivateVdsVDSCommand, log id: 14a8a6ec >2013-03-08 16:05:00,208 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [37206ed6] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:05:00,209 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-42) [37206ed6] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 16:05:00,210 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [37206ed6] Autorecovering 1 storage domains >2013-03-08 16:05:00,210 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [37206ed6] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 16:05:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [1b7b7a84] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:05:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [1b7b7a84] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:05 PM >2013-03-08 16:05:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [1b7b7a84] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:05 PM >2013-03-08 16:05:00,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [423ecb73] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3684aa67 >2013-03-08 16:05:00,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [423ecb73] FINISH, GetHardwareInfoVDSCommand, log id: 3684aa67 >2013-03-08 16:05:00,306 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-54) [2a4019b9] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 16:05:00,309 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-54) [65f788f2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 16:05:00,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 772cada >2013-03-08 16:06:04,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 16:06:04,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:06:04,625 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 16:06:04,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 772cada >2013-03-08 16:06:04,629 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [4613e8eb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:06:04,643 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [4613e8eb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 67e2d42a >2013-03-08 16:06:04,643 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [4613e8eb] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 16:06:04,643 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [4613e8eb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 67e2d42a >2013-03-08 16:06:04,645 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [4613e8eb] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 16:06:04,645 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [65f788f2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 16:06:04,655 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-54) [262726d9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:06:04,656 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [262726d9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 11e9ed8 >2013-03-08 16:06:04,658 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [262726d9] FINISH, SetVdsStatusVDSCommand, log id: 11e9ed8 >2013-03-08 16:06:04,663 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [262726d9] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 16:06:04,686 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [16ffa321] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:06:04,689 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [1d8e03e4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:06:04,690 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-54) [1d8e03e4] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 16:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [64e0fbe6] Autorecovering 1 hosts >2013-03-08 16:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [64e0fbe6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [47a18f7d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [47a18f7d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [47a18f7d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5ef3b692 >2013-03-08 16:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [47a18f7d] FINISH, SetVdsStatusVDSCommand, log id: 5ef3b692 >2013-03-08 16:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [47a18f7d] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4d06cd62 >2013-03-08 16:10:00,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [47a18f7d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 65a96f59 >2013-03-08 16:10:00,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [47a18f7d] FINISH, GetHardwareInfoVDSCommand, log id: 65a96f59 >2013-03-08 16:10:00,243 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-69) [680ea6f8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:10:00,245 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-69) [55a61695] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:10:00,247 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [55a61695] FINISH, ActivateVdsVDSCommand, log id: 4d06cd62 >2013-03-08 16:10:00,266 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [55a61695] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:10:00,267 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-69) [55a61695] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 16:10:00,268 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [55a61695] Autorecovering 1 storage domains >2013-03-08 16:10:00,268 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [55a61695] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 16:10:00,269 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [4afe4c39] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:10:00,269 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [4afe4c39] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:10 PM >2013-03-08 16:10:00,272 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [4afe4c39] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:10 PM >2013-03-08 16:10:01,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [6100460d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3a8c10dd >2013-03-08 16:10:01,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [6100460d] FINISH, GetHardwareInfoVDSCommand, log id: 3a8c10dd >2013-03-08 16:10:01,922 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-9) [1185404b] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 16:10:01,926 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-9) [5824cf4e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 16:10:01,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: a45c8 >2013-03-08 16:11:07,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 16:11:07,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:11:07,929 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 16:11:07,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: a45c8 >2013-03-08 16:11:07,932 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [35afa1a1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:11:07,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [35afa1a1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7e5e1517 >2013-03-08 16:11:07,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [35afa1a1] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 16:11:07,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [35afa1a1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7e5e1517 >2013-03-08 16:11:07,948 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [35afa1a1] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 16:11:07,948 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [5824cf4e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 16:11:07,958 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-9) [6dfaa2ab] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:11:07,960 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [6dfaa2ab] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6324199d >2013-03-08 16:11:07,961 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [6dfaa2ab] FINISH, SetVdsStatusVDSCommand, log id: 6324199d >2013-03-08 16:11:07,967 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-9) [6dfaa2ab] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 16:11:07,989 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-9) [46e2b724] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:11:07,991 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-9) [1799d88f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:11:07,992 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-9) [1799d88f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 16:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [624a23cd] Autorecovering 1 hosts >2013-03-08 16:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [624a23cd] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [3f0d73f6] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [3f0d73f6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [3f0d73f6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 608a4ca5 >2013-03-08 16:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [3f0d73f6] FINISH, SetVdsStatusVDSCommand, log id: 608a4ca5 >2013-03-08 16:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-50) [3f0d73f6] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3f16e011 >2013-03-08 16:15:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [3f0d73f6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 9e3b71d >2013-03-08 16:15:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [3f0d73f6] FINISH, GetHardwareInfoVDSCommand, log id: 9e3b71d >2013-03-08 16:15:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-50) [3802d15f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:15:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-50) [4781c0a4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:15:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-50) [4781c0a4] FINISH, ActivateVdsVDSCommand, log id: 3f16e011 >2013-03-08 16:15:00,219 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [4781c0a4] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:15:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-50) [4781c0a4] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 16:15:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [4781c0a4] Autorecovering 1 storage domains >2013-03-08 16:15:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [4781c0a4] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 16:15:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [4a66fd3e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:15:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [4a66fd3e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:15 PM >2013-03-08 16:15:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [4a66fd3e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:15 PM >2013-03-08 16:15:02,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [79c154e4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 828caae >2013-03-08 16:15:02,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [79c154e4] FINISH, GetHardwareInfoVDSCommand, log id: 828caae >2013-03-08 16:15:02,228 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-25) [2839ca4f] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 16:15:02,232 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-25) [3351855a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 16:15:02,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: db112f5 >2013-03-08 16:16:05,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 16:16:05,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:16:05,214 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 16:16:05,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: db112f5 >2013-03-08 16:16:05,218 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [6b220186] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:16:05,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6b220186] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 21c9da30 >2013-03-08 16:16:05,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [6b220186] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 16:16:05,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6b220186] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 21c9da30 >2013-03-08 16:16:05,234 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [6b220186] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 16:16:05,234 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [3351855a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 16:16:05,243 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-25) [1731a764] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:16:05,245 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [1731a764] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 26777385 >2013-03-08 16:16:05,246 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [1731a764] FINISH, SetVdsStatusVDSCommand, log id: 26777385 >2013-03-08 16:16:05,252 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-25) [1731a764] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 16:16:05,272 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-25) [6e3a829d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:16:05,275 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-25) [d664e39] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:16:05,276 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-25) [d664e39] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 16:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [10761322] Autorecovering 1 hosts >2013-03-08 16:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [10761322] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [4b89d27b] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [4b89d27b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [4b89d27b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 19e9e304 >2013-03-08 16:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [4b89d27b] FINISH, SetVdsStatusVDSCommand, log id: 19e9e304 >2013-03-08 16:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-92) [4b89d27b] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5391cab6 >2013-03-08 16:20:00,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [4b89d27b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 520fb43b >2013-03-08 16:20:00,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [4b89d27b] FINISH, GetHardwareInfoVDSCommand, log id: 520fb43b >2013-03-08 16:20:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-92) [2af6ecc4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:20:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-92) [730fe84f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:20:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-92) [730fe84f] FINISH, ActivateVdsVDSCommand, log id: 5391cab6 >2013-03-08 16:20:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [730fe84f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:20:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-92) [730fe84f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 16:20:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [730fe84f] Autorecovering 1 storage domains >2013-03-08 16:20:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [730fe84f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 16:20:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [4d768856] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:20:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [4d768856] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:20 PM >2013-03-08 16:20:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [4d768856] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:20 PM >2013-03-08 16:20:02,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [1e8ba581] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 788c7e0f >2013-03-08 16:20:02,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [1e8ba581] FINISH, GetHardwareInfoVDSCommand, log id: 788c7e0f >2013-03-08 16:20:02,527 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-77) [4bb1ccac] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 16:20:02,530 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-77) [1f457136] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 16:20:02,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4ffa9fb4 >2013-03-08 16:21:06,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 16:21:06,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:21:06,551 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 16:21:06,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 4ffa9fb4 >2013-03-08 16:21:06,555 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [1c8c800b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:21:06,569 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [1c8c800b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1e05ee12 >2013-03-08 16:21:06,569 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [1c8c800b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 16:21:06,569 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [1c8c800b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1e05ee12 >2013-03-08 16:21:06,571 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [1c8c800b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 16:21:06,571 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [1f457136] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 16:21:06,581 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-77) [41c2212] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:21:06,582 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [41c2212] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1c6b0a5d >2013-03-08 16:21:06,583 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [41c2212] FINISH, SetVdsStatusVDSCommand, log id: 1c6b0a5d >2013-03-08 16:21:06,589 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-77) [41c2212] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 16:21:06,609 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-77) [57a8a28f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:21:06,612 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-77) [52a39daf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:21:06,613 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-77) [52a39daf] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 16:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [30710378] Autorecovering 1 hosts >2013-03-08 16:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [30710378] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-74) [1d1c1456] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-74) [1d1c1456] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:25:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [1d1c1456] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: e787e2 >2013-03-08 16:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [1d1c1456] FINISH, SetVdsStatusVDSCommand, log id: e787e2 >2013-03-08 16:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-74) [1d1c1456] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 74dce201 >2013-03-08 16:25:00,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [1d1c1456] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 54cb4659 >2013-03-08 16:25:00,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [1d1c1456] FINISH, GetHardwareInfoVDSCommand, log id: 54cb4659 >2013-03-08 16:25:00,270 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-74) [929ad16] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:25:00,273 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-74) [60373c67] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:25:00,274 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-74) [60373c67] FINISH, ActivateVdsVDSCommand, log id: 74dce201 >2013-03-08 16:25:00,293 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-74) [60373c67] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:25:00,294 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-74) [60373c67] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 16:25:00,295 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [60373c67] Autorecovering 1 storage domains >2013-03-08 16:25:00,295 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [60373c67] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 16:25:00,296 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-74) [7ff19b05] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:25:00,296 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-74) [7ff19b05] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:25 PM >2013-03-08 16:25:00,299 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-74) [7ff19b05] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:25 PM >2013-03-08 16:25:00,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [3f0da4d7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7be829b >2013-03-08 16:25:00,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [3f0da4d7] FINISH, GetHardwareInfoVDSCommand, log id: 7be829b >2013-03-08 16:25:00,856 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-89) [7dec0d52] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 16:25:00,860 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-89) [b2d478d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 16:25:00,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2eda49 >2013-03-08 16:26:05,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 16:26:05,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:26:05,908 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 16:26:05,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2eda49 >2013-03-08 16:26:05,911 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [7c6a16b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:26:05,925 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [7c6a16b5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3b31847a >2013-03-08 16:26:05,925 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [7c6a16b5] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 16:26:05,925 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [7c6a16b5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3b31847a >2013-03-08 16:26:05,927 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [7c6a16b5] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 16:26:05,928 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [b2d478d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 16:26:05,937 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-89) [313ec60c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:26:05,939 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [313ec60c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 410f1f5d >2013-03-08 16:26:05,940 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [313ec60c] FINISH, SetVdsStatusVDSCommand, log id: 410f1f5d >2013-03-08 16:26:05,946 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-89) [313ec60c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 16:26:05,971 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-89) [2616fdbb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:26:05,973 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-89) [34cfa5f2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:26:05,974 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-89) [34cfa5f2] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 16:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-66) [4378fd6] Autorecovering 1 hosts >2013-03-08 16:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-66) [4378fd6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-66) [62d62a6a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-66) [62d62a6a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [62d62a6a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6c951653 >2013-03-08 16:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [62d62a6a] FINISH, SetVdsStatusVDSCommand, log id: 6c951653 >2013-03-08 16:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-66) [62d62a6a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7820255e >2013-03-08 16:30:00,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [62d62a6a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2190153 >2013-03-08 16:30:00,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [62d62a6a] FINISH, GetHardwareInfoVDSCommand, log id: 2190153 >2013-03-08 16:30:00,288 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-66) [64ea6810] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:30:00,291 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-66) [13a2c9e8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:30:00,292 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-66) [13a2c9e8] FINISH, ActivateVdsVDSCommand, log id: 7820255e >2013-03-08 16:30:00,311 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-66) [13a2c9e8] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:30:00,312 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-66) [13a2c9e8] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 16:30:00,314 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-66) [13a2c9e8] Autorecovering 1 storage domains >2013-03-08 16:30:00,314 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-66) [13a2c9e8] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 16:30:00,314 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-66) [903e742] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:30:00,314 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-66) [903e742] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:30 PM >2013-03-08 16:30:00,317 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-66) [903e742] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:30 PM >2013-03-08 16:30:00,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [3ca1cd17] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 186b682c >2013-03-08 16:30:00,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [3ca1cd17] FINISH, GetHardwareInfoVDSCommand, log id: 186b682c >2013-03-08 16:30:00,390 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-84) [209801ff] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 16:30:00,393 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-84) [46fb169] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 16:30:00,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 29c85a1 >2013-03-08 16:30:33,747 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-27) [7e7f6470] Start refreshing all users data >2013-03-08 16:30:34,698 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Attempting to update VMs/Templates Ovf. >2013-03-08 16:30:34,699 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 16:30:34,700 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 16:30:34,700 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 16:30:34,701 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 16:30:34,701 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 16:30:34,701 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 16:31:04,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 16:31:04,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:31:04,387 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 16:31:04,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 29c85a1 >2013-03-08 16:31:04,390 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [774aab3f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:31:04,404 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [774aab3f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 60e17f45 >2013-03-08 16:31:04,404 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [774aab3f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 16:31:04,404 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [774aab3f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 60e17f45 >2013-03-08 16:31:04,406 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [774aab3f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 16:31:04,407 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [46fb169] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 16:31:04,416 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-84) [7d747abd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:31:04,418 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [7d747abd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5322f135 >2013-03-08 16:31:04,419 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [7d747abd] FINISH, SetVdsStatusVDSCommand, log id: 5322f135 >2013-03-08 16:31:04,425 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-84) [7d747abd] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 16:31:04,447 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-84) [119d6c98] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:31:04,449 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-84) [7c0a6aba] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:31:04,450 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-84) [7c0a6aba] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 16:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [31bb714e] Autorecovering 1 hosts >2013-03-08 16:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [31bb714e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [3942ec60] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [3942ec60] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [3942ec60] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5f551821 >2013-03-08 16:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [3942ec60] FINISH, SetVdsStatusVDSCommand, log id: 5f551821 >2013-03-08 16:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [3942ec60] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 60b52a3c >2013-03-08 16:35:00,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [3942ec60] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 46ea6d97 >2013-03-08 16:35:00,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [3942ec60] FINISH, GetHardwareInfoVDSCommand, log id: 46ea6d97 >2013-03-08 16:35:00,490 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [1dd2cd9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:35:00,493 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [14063627] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:35:00,494 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [14063627] FINISH, ActivateVdsVDSCommand, log id: 60b52a3c >2013-03-08 16:35:00,513 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [14063627] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:35:00,514 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [14063627] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 16:35:00,515 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [14063627] Autorecovering 1 storage domains >2013-03-08 16:35:00,515 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [14063627] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 16:35:00,516 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [288cf14a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:35:00,516 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [288cf14a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:35 PM >2013-03-08 16:35:00,519 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [288cf14a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:35 PM >2013-03-08 16:35:01,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [6a49e344] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 43eb4965 >2013-03-08 16:35:01,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [6a49e344] FINISH, GetHardwareInfoVDSCommand, log id: 43eb4965 >2013-03-08 16:35:01,692 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-2) [9725d04] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 16:35:01,696 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-2) [52676db9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 16:35:01,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 75e90dd7 >2013-03-08 16:36:06,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 16:36:06,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:36:06,700 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 16:36:06,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 75e90dd7 >2013-03-08 16:36:06,703 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [502fbb08] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:36:06,726 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [502fbb08] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1a8ba9e7 >2013-03-08 16:36:06,726 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [502fbb08] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 16:36:06,726 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [502fbb08] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1a8ba9e7 >2013-03-08 16:36:06,728 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [502fbb08] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 16:36:06,729 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [52676db9] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 16:36:06,738 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-2) [6f476a24] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:36:06,740 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [6f476a24] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 35341660 >2013-03-08 16:36:06,741 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [6f476a24] FINISH, SetVdsStatusVDSCommand, log id: 35341660 >2013-03-08 16:36:06,748 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [6f476a24] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 16:36:06,772 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [6697b610] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:36:06,774 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [3d58d304] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:36:06,776 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-2) [3d58d304] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 16:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [39b93541] Autorecovering 1 hosts >2013-03-08 16:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [39b93541] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:40:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [79531718] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [79531718] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [79531718] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 348d6524 >2013-03-08 16:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [79531718] FINISH, SetVdsStatusVDSCommand, log id: 348d6524 >2013-03-08 16:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [79531718] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 799c0d12 >2013-03-08 16:40:00,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [79531718] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 136388d7 >2013-03-08 16:40:00,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [79531718] FINISH, GetHardwareInfoVDSCommand, log id: 136388d7 >2013-03-08 16:40:00,086 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-14) [69f57dd5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:40:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-14) [5bd009de] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:40:00,090 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [5bd009de] FINISH, ActivateVdsVDSCommand, log id: 799c0d12 >2013-03-08 16:40:00,109 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [5bd009de] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:40:00,110 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-14) [5bd009de] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 16:40:00,112 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [5bd009de] Autorecovering 1 storage domains >2013-03-08 16:40:00,112 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [5bd009de] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 16:40:00,112 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [16d33484] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:40:00,113 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [16d33484] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:40 PM >2013-03-08 16:40:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [16d33484] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:40 PM >2013-03-08 16:40:00,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [2a4ad98c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 33f88921 >2013-03-08 16:40:00,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [2a4ad98c] FINISH, GetHardwareInfoVDSCommand, log id: 33f88921 >2013-03-08 16:40:01,019 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-26) [4217a8ae] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 16:40:01,023 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-26) [505ef456] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 16:40:01,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 60c12952 >2013-03-08 16:41:06,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 16:41:06,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:41:06,072 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 16:41:06,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 60c12952 >2013-03-08 16:41:06,076 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [267cda98] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:41:06,090 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [267cda98] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7beb124d >2013-03-08 16:41:06,090 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [267cda98] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 16:41:06,090 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [267cda98] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7beb124d >2013-03-08 16:41:06,092 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [267cda98] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 16:41:06,092 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [505ef456] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 16:41:06,103 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-26) [6f6d55ee] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:41:06,104 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [6f6d55ee] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 51bfa752 >2013-03-08 16:41:06,105 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [6f6d55ee] FINISH, SetVdsStatusVDSCommand, log id: 51bfa752 >2013-03-08 16:41:06,112 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [6f6d55ee] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 16:41:06,135 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [5be245f5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:41:06,138 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [6da5caa3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:41:06,139 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-26) [6da5caa3] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 16:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [497e931f] Autorecovering 1 hosts >2013-03-08 16:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [497e931f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [172dc1a4] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [172dc1a4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [172dc1a4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 394a6a43 >2013-03-08 16:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [172dc1a4] FINISH, SetVdsStatusVDSCommand, log id: 394a6a43 >2013-03-08 16:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [172dc1a4] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1f77ab20 >2013-03-08 16:45:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [172dc1a4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1427f500 >2013-03-08 16:45:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [172dc1a4] FINISH, GetHardwareInfoVDSCommand, log id: 1427f500 >2013-03-08 16:45:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [45f6b4a6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:45:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [283911ed] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:45:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [283911ed] FINISH, ActivateVdsVDSCommand, log id: 1f77ab20 >2013-03-08 16:45:00,221 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [283911ed] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:45:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [283911ed] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 16:45:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [283911ed] Autorecovering 1 storage domains >2013-03-08 16:45:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [283911ed] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 16:45:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [3b862665] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:45:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [3b862665] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:45 PM >2013-03-08 16:45:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [3b862665] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:45 PM >2013-03-08 16:45:00,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [2fa45b6a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 19942af5 >2013-03-08 16:45:00,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [2fa45b6a] FINISH, GetHardwareInfoVDSCommand, log id: 19942af5 >2013-03-08 16:45:00,382 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-90) [75967a2d] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 16:45:00,386 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-90) [165be0f2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 16:45:00,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 298ccb4e >2013-03-08 16:46:04,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 16:46:04,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:46:04,698 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 16:46:04,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 298ccb4e >2013-03-08 16:46:04,702 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [67c0049a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:46:04,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [67c0049a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3a8364ea >2013-03-08 16:46:04,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [67c0049a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 16:46:04,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [67c0049a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3a8364ea >2013-03-08 16:46:04,726 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [67c0049a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 16:46:04,726 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [165be0f2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 16:46:04,736 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-90) [217fd96d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:46:04,737 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [217fd96d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7dc74142 >2013-03-08 16:46:04,738 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [217fd96d] FINISH, SetVdsStatusVDSCommand, log id: 7dc74142 >2013-03-08 16:46:04,744 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [217fd96d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 16:46:04,764 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-90) [14ed86cb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:46:04,766 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-90) [727ae14e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:46:04,768 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-90) [727ae14e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 16:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [61ca503a] Autorecovering 1 hosts >2013-03-08 16:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [61ca503a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [3d9ea699] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [3d9ea699] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [3d9ea699] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7f9d8899 >2013-03-08 16:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [3d9ea699] FINISH, SetVdsStatusVDSCommand, log id: 7f9d8899 >2013-03-08 16:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-1) [3d9ea699] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6e27d058 >2013-03-08 16:50:00,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [3d9ea699] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 70a57d41 >2013-03-08 16:50:00,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [3d9ea699] FINISH, GetHardwareInfoVDSCommand, log id: 70a57d41 >2013-03-08 16:50:00,410 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-1) [3eed2c6a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:50:00,413 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-1) [58891fab] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:50:00,414 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-1) [58891fab] FINISH, ActivateVdsVDSCommand, log id: 6e27d058 >2013-03-08 16:50:00,433 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [58891fab] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:50:00,434 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-1) [58891fab] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 16:50:00,435 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [58891fab] Autorecovering 1 storage domains >2013-03-08 16:50:00,436 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [58891fab] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 16:50:00,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [7228cbd3] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:50:00,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [7228cbd3] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:50 PM >2013-03-08 16:50:00,439 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [7228cbd3] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:50 PM >2013-03-08 16:50:01,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [2323f6cd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 242b0432 >2013-03-08 16:50:01,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [2323f6cd] FINISH, GetHardwareInfoVDSCommand, log id: 242b0432 >2013-03-08 16:50:02,014 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-79) [403e7ad7] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 16:50:02,017 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-79) [616d5c2f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 16:50:02,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 60972f96 >2013-03-08 16:51:06,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 16:51:06,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:51:06,996 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 16:51:06,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 60972f96 >2013-03-08 16:51:07,000 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [579d835f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:51:07,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [579d835f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5dc253fd >2013-03-08 16:51:07,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [579d835f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 16:51:07,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [579d835f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5dc253fd >2013-03-08 16:51:07,016 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [579d835f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-08 16:51:07,016 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [616d5c2f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 16:51:07,026 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-79) [6b33afd9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:51:07,028 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [6b33afd9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 565618a5 >2013-03-08 16:51:07,029 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [6b33afd9] FINISH, SetVdsStatusVDSCommand, log id: 565618a5 >2013-03-08 16:51:07,035 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-79) [6b33afd9] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 16:51:07,056 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-79) [13573ac] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:51:07,058 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-79) [770cf360] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:51:07,060 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-79) [770cf360] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 16:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [f349d04] Autorecovering 1 hosts >2013-03-08 16:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [f349d04] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-67) [40055233] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-67) [40055233] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [40055233] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 621096d1 >2013-03-08 16:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [40055233] FINISH, SetVdsStatusVDSCommand, log id: 621096d1 >2013-03-08 16:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-67) [40055233] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6f4fa2d3 >2013-03-08 16:55:00,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [40055233] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 554d4779 >2013-03-08 16:55:00,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [40055233] FINISH, GetHardwareInfoVDSCommand, log id: 554d4779 >2013-03-08 16:55:00,267 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-67) [68f9305] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:55:00,270 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-67) [37e2851e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:55:00,271 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-67) [37e2851e] FINISH, ActivateVdsVDSCommand, log id: 6f4fa2d3 >2013-03-08 16:55:00,290 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-67) [37e2851e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 16:55:00,291 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-67) [37e2851e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 16:55:00,292 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [37e2851e] Autorecovering 1 storage domains >2013-03-08 16:55:00,292 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [37e2851e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 16:55:00,293 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-67) [2d511931] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:55:00,293 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-67) [2d511931] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 4:55 PM >2013-03-08 16:55:00,296 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-67) [2d511931] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 4:55 PM >2013-03-08 16:55:01,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [3bfb99c0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5f345d70 >2013-03-08 16:55:01,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [3bfb99c0] FINISH, GetHardwareInfoVDSCommand, log id: 5f345d70 >2013-03-08 16:55:01,307 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-33) [b4f1d6d] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 16:55:01,310 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-33) [4038a880] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 16:55:01,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1e612b80 >2013-03-08 16:56:07,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-08 16:56:07,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-08 16:56:07,677 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-08 16:56:07,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 1e612b80 >2013-03-08 16:56:07,680 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [36cc831d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 16:56:07,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [36cc831d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1d55ca1e >2013-03-08 16:56:07,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [36cc831d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-08 16:56:07,697 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [36cc831d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1d55ca1e >2013-03-08 16:56:07,701 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [4038a880] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-08 16:56:07,711 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-33) [2562d874] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:56:07,713 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [2562d874] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7094329f >2013-03-08 16:56:07,714 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [2562d874] FINISH, SetVdsStatusVDSCommand, log id: 7094329f >2013-03-08 16:56:07,720 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-33) [2562d874] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-08 16:56:07,744 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-33) [57c81d81] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:56:07,746 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-33) [68a361a0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 16:56:07,748 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-33) [68a361a0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-08 17:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) [3fbbb1fc] Autorecovering 1 hosts >2013-03-08 17:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) [3fbbb1fc] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-08 17:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-30) [3695e22c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 17:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-30) [3695e22c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 17:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-30) [3695e22c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 733e921e >2013-03-08 17:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-30) [3695e22c] FINISH, SetVdsStatusVDSCommand, log id: 733e921e >2013-03-08 17:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-30) [3695e22c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: d07cc9 >2013-03-08 17:00:00,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-30) [3695e22c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 78853b51 >2013-03-08 17:00:00,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-30) [3695e22c] FINISH, GetHardwareInfoVDSCommand, log id: 78853b51 >2013-03-08 17:00:00,237 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-30) [4e8f8c53] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 17:00:00,239 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-30) [6209d74c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 17:00:00,240 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-30) [6209d74c] FINISH, ActivateVdsVDSCommand, log id: d07cc9 >2013-03-08 17:00:00,260 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-30) [6209d74c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-08 17:00:00,261 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-30) [6209d74c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-08 17:00:00,263 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) [6209d74c] Autorecovering 1 storage domains >2013-03-08 17:00:00,263 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) [6209d74c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-08 17:00:00,263 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-30) [4e5c16e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-08 17:00:00,264 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-30) [4e5c16e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/8/13 5:00 PM >2013-03-08 17:00:00,266 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-30) [4e5c16e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/8/13 5:00 PM >2013-03-08 17:00:01,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [1c4e755b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 214b9ad5 >2013-03-08 17:00:01,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [1c4e755b] FINISH, GetHardwareInfoVDSCommand, log id: 214b9ad5 >2013-03-08 17:00:01,986 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-38) [56746abd] Running command: InitVdsOnUpCommand internal: true. >2013-03-08 17:00:01,989 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-38) [425587f4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-08 17:00:01,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7ff28065 >2013-03-08 17:00:07,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 7ff28065 >2013-03-08 17:00:07,875 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-50) [425587f4] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2013-03-08 17:00:07,878 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-38) [a4ee4af] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 17:00:07,883 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-38) [5d712191] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-08 17:00:08,370 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [1ce1a6db] hostFromVds::selectedVds - cheetah02.scl.lab.tlv.redhat.com, spmStatus Free, storage pool Default >2013-03-08 17:00:08,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [1ce1a6db] SpmStatus on vds 5e9ad544-fe9f-4669-8b11-1631ba7d1c12: Free >2013-03-08 17:00:08,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [1ce1a6db] starting spm on vds cheetah02.scl.lab.tlv.redhat.com, storage pool Default, prevId 2, LVER 0 >2013-03-08 17:00:08,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [1ce1a6db] START, SpmStartVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, prevId=2, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3ca57e60 >2013-03-08 17:00:08,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [1ce1a6db] spmStart polling started: taskId = cf837558-58c7-4568-83a4-9a235ccd0887 >2013-03-08 17:00:29,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [1ce1a6db] spmStart polling ended: taskId = cf837558-58c7-4568-83a4-9a235ccd0887 task status = finished >2013-03-08 17:00:29,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [1ce1a6db] spmStart polling ended, spm status: SPM >2013-03-08 17:00:29,935 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [1ce1a6db] START, HSMClearTaskVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, taskId=cf837558-58c7-4568-83a4-9a235ccd0887), log id: 2bd0db93 >2013-03-08 17:00:29,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (QuartzScheduler_Worker-91) [1ce1a6db] FINISH, HSMClearTaskVDSCommand, log id: 2bd0db93 >2013-03-08 17:00:29,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (QuartzScheduler_Worker-91) [1ce1a6db] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1cffa65e, log id: 3ca57e60 >2013-03-08 17:00:29,958 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-91) [1ce1a6db] Initialize Irs proxy from vds: cheetah02.scl.lab.tlv.redhat.com >2013-03-08 17:00:41,254 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-56) [d1e2a77] Storage Domain 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126:D1 was reported by Host cheetah02.scl.lab.tlv.redhat.com as Active in Pool 5849b030-626e-47cb-ad90-3ce782d831b3, moving to active status >2013-03-08 17:30:33,748 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-59) [2aee83e2] Start refreshing all users data >2013-03-08 17:30:34,701 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) [1d8e03e4] Attempting to update VMs/Templates Ovf. >2013-03-08 17:30:34,702 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) [1d8e03e4] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 17:30:34,702 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) [1d8e03e4] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 17:30:34,703 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) [1d8e03e4] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 17:30:34,703 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) [1d8e03e4] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 17:30:34,703 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) [1d8e03e4] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 17:30:34,704 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) [1d8e03e4] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 17:30:34,704 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) [1d8e03e4] Attempting to update VM OVFs in Data Center Default >2013-03-08 17:30:34,705 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) [1d8e03e4] Successfully updated VM OVFs in Data Center Default >2013-03-08 17:30:34,705 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) [1d8e03e4] Attempting to update template OVFs in Data Center Default >2013-03-08 17:30:34,705 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) [1d8e03e4] Succesfully updated templates OVFs in Data Center Default >2013-03-08 17:30:34,705 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) [1d8e03e4] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-08 17:30:34,706 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-54) [1d8e03e4] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-08 18:30:33,750 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-31) [310fbd06] Start refreshing all users data >2013-03-08 18:30:34,706 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-82) [68b8a2e0] Attempting to update VMs/Templates Ovf. >2013-03-08 18:30:34,708 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-82) [68b8a2e0] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 18:30:34,708 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-82) [68b8a2e0] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 18:30:34,709 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-82) [68b8a2e0] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 18:30:34,709 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-82) [68b8a2e0] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 18:30:34,709 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-82) [68b8a2e0] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 18:30:34,710 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-82) [68b8a2e0] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 18:30:34,710 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-82) [68b8a2e0] Attempting to update VM OVFs in Data Center Default >2013-03-08 18:30:34,711 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-82) [68b8a2e0] Successfully updated VM OVFs in Data Center Default >2013-03-08 18:30:34,711 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-82) [68b8a2e0] Attempting to update template OVFs in Data Center Default >2013-03-08 18:30:34,712 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-82) [68b8a2e0] Succesfully updated templates OVFs in Data Center Default >2013-03-08 18:30:34,712 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-82) [68b8a2e0] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-08 18:30:34,712 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-82) [68b8a2e0] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-08 19:30:33,752 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-82) [68b8a2e0] Start refreshing all users data >2013-03-08 19:30:34,712 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Attempting to update VMs/Templates Ovf. >2013-03-08 19:30:34,713 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 19:30:34,714 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 19:30:34,714 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 19:30:34,715 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 19:30:34,715 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 19:30:34,715 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 19:30:34,715 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Attempting to update VM OVFs in Data Center Default >2013-03-08 19:30:34,716 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Successfully updated VM OVFs in Data Center Default >2013-03-08 19:30:34,716 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Attempting to update template OVFs in Data Center Default >2013-03-08 19:30:34,717 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Succesfully updated templates OVFs in Data Center Default >2013-03-08 19:30:34,717 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-08 19:30:34,717 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [2bb2d7fd] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-08 20:30:33,754 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-86) [5c61f6bb] Start refreshing all users data >2013-03-08 20:30:34,718 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [79bd4981] Attempting to update VMs/Templates Ovf. >2013-03-08 20:30:34,719 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [79bd4981] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 20:30:34,719 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [79bd4981] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 20:30:34,720 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [79bd4981] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 20:30:34,720 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [79bd4981] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 20:30:34,720 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [79bd4981] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 20:30:34,721 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [79bd4981] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 20:30:34,721 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [79bd4981] Attempting to update VM OVFs in Data Center Default >2013-03-08 20:30:34,722 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [79bd4981] Successfully updated VM OVFs in Data Center Default >2013-03-08 20:30:34,722 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [79bd4981] Attempting to update template OVFs in Data Center Default >2013-03-08 20:30:34,722 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [79bd4981] Succesfully updated templates OVFs in Data Center Default >2013-03-08 20:30:34,722 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [79bd4981] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-08 20:30:34,723 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [79bd4981] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-08 21:30:33,757 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-65) [617179c6] Start refreshing all users data >2013-03-08 21:30:34,723 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-7) [463f5475] Attempting to update VMs/Templates Ovf. >2013-03-08 21:30:34,725 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-7) [463f5475] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 21:30:34,725 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-7) [463f5475] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 21:30:34,725 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-7) [463f5475] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 21:30:34,726 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-7) [463f5475] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 21:30:34,726 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-7) [463f5475] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 21:30:34,727 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-7) [463f5475] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 21:30:34,727 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-7) [463f5475] Attempting to update VM OVFs in Data Center Default >2013-03-08 21:30:34,727 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-7) [463f5475] Successfully updated VM OVFs in Data Center Default >2013-03-08 21:30:34,728 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-7) [463f5475] Attempting to update template OVFs in Data Center Default >2013-03-08 21:30:34,728 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-7) [463f5475] Succesfully updated templates OVFs in Data Center Default >2013-03-08 21:30:34,728 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-7) [463f5475] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-08 21:30:34,729 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-7) [463f5475] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-08 22:30:33,759 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-67) [2d511931] Start refreshing all users data >2013-03-08 22:30:34,730 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [288cf14a] Attempting to update VMs/Templates Ovf. >2013-03-08 22:30:34,731 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [288cf14a] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 22:30:34,738 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [288cf14a] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 22:30:34,738 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [288cf14a] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 22:30:34,740 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [288cf14a] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 22:30:34,740 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [288cf14a] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 22:30:34,741 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [288cf14a] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 22:30:34,741 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [288cf14a] Attempting to update VM OVFs in Data Center Default >2013-03-08 22:30:34,742 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [288cf14a] Successfully updated VM OVFs in Data Center Default >2013-03-08 22:30:34,742 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [288cf14a] Attempting to update template OVFs in Data Center Default >2013-03-08 22:30:34,743 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [288cf14a] Succesfully updated templates OVFs in Data Center Default >2013-03-08 22:30:34,743 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [288cf14a] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-08 22:30:34,744 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [288cf14a] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-08 23:30:33,761 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-67) [2d511931] Start refreshing all users data >2013-03-08 23:30:34,745 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [3113df82] Attempting to update VMs/Templates Ovf. >2013-03-08 23:30:34,746 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [3113df82] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 23:30:34,753 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [3113df82] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 23:30:34,753 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [3113df82] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 23:30:34,755 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [3113df82] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 23:30:34,755 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [3113df82] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 23:30:34,756 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [3113df82] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-08 23:30:34,756 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [3113df82] Attempting to update VM OVFs in Data Center Default >2013-03-08 23:30:34,757 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [3113df82] Successfully updated VM OVFs in Data Center Default >2013-03-08 23:30:34,757 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [3113df82] Attempting to update template OVFs in Data Center Default >2013-03-08 23:30:34,758 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [3113df82] Succesfully updated templates OVFs in Data Center Default >2013-03-08 23:30:34,758 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [3113df82] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-08 23:30:34,758 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [3113df82] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 00:30:33,763 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-24) [3113df82] Start refreshing all users data >2013-03-09 00:30:34,759 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-6) [14e3210e] Attempting to update VMs/Templates Ovf. >2013-03-09 00:30:34,761 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-6) [14e3210e] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 00:30:34,761 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-6) [14e3210e] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 00:30:34,762 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-6) [14e3210e] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 00:30:34,762 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-6) [14e3210e] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 00:30:34,762 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-6) [14e3210e] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 00:30:34,763 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-6) [14e3210e] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 00:30:34,763 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-6) [14e3210e] Attempting to update VM OVFs in Data Center Default >2013-03-09 00:30:34,764 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-6) [14e3210e] Successfully updated VM OVFs in Data Center Default >2013-03-09 00:30:34,764 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-6) [14e3210e] Attempting to update template OVFs in Data Center Default >2013-03-09 00:30:34,764 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-6) [14e3210e] Succesfully updated templates OVFs in Data Center Default >2013-03-09 00:30:34,765 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-6) [14e3210e] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 00:30:34,765 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-6) [14e3210e] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 01:30:33,765 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-6) [14e3210e] Start refreshing all users data >2013-03-09 01:30:34,765 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-79) [770cf360] Attempting to update VMs/Templates Ovf. >2013-03-09 01:30:34,766 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-79) [770cf360] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 01:30:34,767 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-79) [770cf360] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 01:30:34,767 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-79) [770cf360] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 01:30:34,768 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-79) [770cf360] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 01:30:34,768 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-79) [770cf360] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 01:30:34,768 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-79) [770cf360] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 01:30:34,768 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-79) [770cf360] Attempting to update VM OVFs in Data Center Default >2013-03-09 01:30:34,769 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-79) [770cf360] Successfully updated VM OVFs in Data Center Default >2013-03-09 01:30:34,769 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-79) [770cf360] Attempting to update template OVFs in Data Center Default >2013-03-09 01:30:34,770 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-79) [770cf360] Succesfully updated templates OVFs in Data Center Default >2013-03-09 01:30:34,770 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-79) [770cf360] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 01:30:34,770 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-79) [770cf360] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 02:30:33,767 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-79) [770cf360] Start refreshing all users data >2013-03-09 02:30:34,771 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2ff6ef51] Attempting to update VMs/Templates Ovf. >2013-03-09 02:30:34,772 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2ff6ef51] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 02:30:34,773 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2ff6ef51] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 02:30:34,773 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2ff6ef51] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 02:30:34,773 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2ff6ef51] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 02:30:34,773 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2ff6ef51] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 02:30:34,774 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2ff6ef51] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 02:30:34,774 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2ff6ef51] Attempting to update VM OVFs in Data Center Default >2013-03-09 02:30:34,775 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2ff6ef51] Successfully updated VM OVFs in Data Center Default >2013-03-09 02:30:34,775 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2ff6ef51] Attempting to update template OVFs in Data Center Default >2013-03-09 02:30:34,775 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2ff6ef51] Succesfully updated templates OVFs in Data Center Default >2013-03-09 02:30:34,775 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2ff6ef51] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 02:30:34,776 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-81) [2ff6ef51] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 03:30:33,769 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-81) [2ff6ef51] Start refreshing all users data >2013-03-09 03:30:34,777 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Attempting to update VMs/Templates Ovf. >2013-03-09 03:30:34,778 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 03:30:34,779 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 03:30:34,779 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 03:30:34,780 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 03:30:34,780 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 03:30:34,780 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 03:30:34,780 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Attempting to update VM OVFs in Data Center Default >2013-03-09 03:30:34,781 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Successfully updated VM OVFs in Data Center Default >2013-03-09 03:30:34,781 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Attempting to update template OVFs in Data Center Default >2013-03-09 03:30:34,782 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Succesfully updated templates OVFs in Data Center Default >2013-03-09 03:30:34,782 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 03:30:34,782 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (QuartzScheduler_Worker-29) [2bb2d7fd] AuditLogCleanupManager::deleteAgedOutAuditLogs - entered >2013-03-09 04:30:33,771 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-93) [cf9e7f6] Start refreshing all users data >2013-03-09 04:30:34,783 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-73) [10e3e76a] Attempting to update VMs/Templates Ovf. >2013-03-09 04:30:34,784 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-73) [10e3e76a] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 04:30:34,785 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-73) [10e3e76a] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 04:30:34,785 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-73) [10e3e76a] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 04:30:34,785 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-73) [10e3e76a] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 04:30:34,785 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-73) [10e3e76a] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 04:30:34,786 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-73) [10e3e76a] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 04:30:34,786 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-73) [10e3e76a] Attempting to update VM OVFs in Data Center Default >2013-03-09 04:30:34,787 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-73) [10e3e76a] Successfully updated VM OVFs in Data Center Default >2013-03-09 04:30:34,787 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-73) [10e3e76a] Attempting to update template OVFs in Data Center Default >2013-03-09 04:30:34,787 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-73) [10e3e76a] Succesfully updated templates OVFs in Data Center Default >2013-03-09 04:30:34,787 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-73) [10e3e76a] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 04:30:34,788 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-73) [10e3e76a] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 05:30:33,774 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-42) [1b7b7a84] Start refreshing all users data >2013-03-09 05:30:34,789 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-60) [28f6d5ca] Attempting to update VMs/Templates Ovf. >2013-03-09 05:30:34,790 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-60) [28f6d5ca] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 05:30:34,790 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-60) [28f6d5ca] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 05:30:34,790 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-60) [28f6d5ca] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 05:30:34,791 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-60) [28f6d5ca] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 05:30:34,791 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-60) [28f6d5ca] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 05:30:34,792 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-60) [28f6d5ca] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 05:30:34,792 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-60) [28f6d5ca] Attempting to update VM OVFs in Data Center Default >2013-03-09 05:30:34,792 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-60) [28f6d5ca] Successfully updated VM OVFs in Data Center Default >2013-03-09 05:30:34,793 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-60) [28f6d5ca] Attempting to update template OVFs in Data Center Default >2013-03-09 05:30:34,793 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-60) [28f6d5ca] Succesfully updated templates OVFs in Data Center Default >2013-03-09 05:30:34,793 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-60) [28f6d5ca] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 05:30:34,794 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-60) [28f6d5ca] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 06:30:33,777 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-9) [1799d88f] Start refreshing all users data >2013-03-09 06:30:34,795 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Attempting to update VMs/Templates Ovf. >2013-03-09 06:30:34,796 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 06:30:34,796 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 06:30:34,796 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 06:30:34,797 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 06:30:34,797 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 06:30:34,798 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 06:30:34,798 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Attempting to update VM OVFs in Data Center Default >2013-03-09 06:30:34,799 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Successfully updated VM OVFs in Data Center Default >2013-03-09 06:30:34,799 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Attempting to update template OVFs in Data Center Default >2013-03-09 06:30:34,799 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Succesfully updated templates OVFs in Data Center Default >2013-03-09 06:30:34,799 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 06:30:34,800 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 07:30:33,778 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-98) [37e37665] Start refreshing all users data >2013-03-09 07:30:34,800 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) [57ef1896] Attempting to update VMs/Templates Ovf. >2013-03-09 07:30:34,801 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) [57ef1896] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 07:30:34,802 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) [57ef1896] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 07:30:34,802 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) [57ef1896] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 07:30:34,803 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) [57ef1896] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 07:30:34,803 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) [57ef1896] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 07:30:34,803 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) [57ef1896] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 07:30:34,803 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) [57ef1896] Attempting to update VM OVFs in Data Center Default >2013-03-09 07:30:34,804 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) [57ef1896] Successfully updated VM OVFs in Data Center Default >2013-03-09 07:30:34,804 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) [57ef1896] Attempting to update template OVFs in Data Center Default >2013-03-09 07:30:34,805 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) [57ef1896] Succesfully updated templates OVFs in Data Center Default >2013-03-09 07:30:34,805 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) [57ef1896] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 07:30:34,805 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-71) [57ef1896] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 08:30:33,779 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-40) [90e1ee3] Start refreshing all users data >2013-03-09 08:30:34,806 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-99) [42313555] Attempting to update VMs/Templates Ovf. >2013-03-09 08:30:34,807 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-99) [42313555] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 08:30:34,807 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-99) [42313555] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 08:30:34,808 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-99) [42313555] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 08:30:34,808 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-99) [42313555] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 08:30:34,808 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-99) [42313555] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 08:30:34,809 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-99) [42313555] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 08:30:34,809 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-99) [42313555] Attempting to update VM OVFs in Data Center Default >2013-03-09 08:30:34,809 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-99) [42313555] Successfully updated VM OVFs in Data Center Default >2013-03-09 08:30:34,810 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-99) [42313555] Attempting to update template OVFs in Data Center Default >2013-03-09 08:30:34,810 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-99) [42313555] Succesfully updated templates OVFs in Data Center Default >2013-03-09 08:30:34,810 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-99) [42313555] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 08:30:34,811 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-99) [42313555] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 09:30:33,781 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-10) [3719a4cc] Start refreshing all users data >2013-03-09 09:30:34,811 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-15) [6ca6376] Attempting to update VMs/Templates Ovf. >2013-03-09 09:30:34,812 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-15) [6ca6376] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 09:30:34,813 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-15) [6ca6376] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 09:30:34,813 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-15) [6ca6376] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 09:30:34,813 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-15) [6ca6376] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 09:30:34,813 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-15) [6ca6376] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 09:30:34,814 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-15) [6ca6376] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 09:30:34,814 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-15) [6ca6376] Attempting to update VM OVFs in Data Center Default >2013-03-09 09:30:34,815 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-15) [6ca6376] Successfully updated VM OVFs in Data Center Default >2013-03-09 09:30:34,815 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-15) [6ca6376] Attempting to update template OVFs in Data Center Default >2013-03-09 09:30:34,815 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-15) [6ca6376] Succesfully updated templates OVFs in Data Center Default >2013-03-09 09:30:34,815 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-15) [6ca6376] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 09:30:34,816 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-15) [6ca6376] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 10:30:33,783 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-15) [6ca6376] Start refreshing all users data >2013-03-09 10:30:34,816 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Attempting to update VMs/Templates Ovf. >2013-03-09 10:30:34,817 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 10:30:34,818 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 10:30:34,818 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 10:30:34,818 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 10:30:34,818 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 10:30:34,819 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 10:30:34,819 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Attempting to update VM OVFs in Data Center Default >2013-03-09 10:30:34,820 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Successfully updated VM OVFs in Data Center Default >2013-03-09 10:30:34,820 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Attempting to update template OVFs in Data Center Default >2013-03-09 10:30:34,820 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Succesfully updated templates OVFs in Data Center Default >2013-03-09 10:30:34,821 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 10:30:34,821 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 11:30:33,786 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-96) [23ffeabc] Start refreshing all users data >2013-03-09 11:30:34,821 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [6edef405] Attempting to update VMs/Templates Ovf. >2013-03-09 11:30:34,822 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [6edef405] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 11:30:34,823 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [6edef405] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 11:30:34,823 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [6edef405] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 11:30:34,824 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [6edef405] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 11:30:34,824 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [6edef405] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 11:30:34,824 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [6edef405] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 11:30:34,824 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [6edef405] Attempting to update VM OVFs in Data Center Default >2013-03-09 11:30:34,825 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [6edef405] Successfully updated VM OVFs in Data Center Default >2013-03-09 11:30:34,825 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [6edef405] Attempting to update template OVFs in Data Center Default >2013-03-09 11:30:34,826 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [6edef405] Succesfully updated templates OVFs in Data Center Default >2013-03-09 11:30:34,826 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [6edef405] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 11:30:34,826 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [6edef405] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 12:30:33,788 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-58) [ee9c6e6] Start refreshing all users data >2013-03-09 12:30:34,827 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) [368aa39f] Attempting to update VMs/Templates Ovf. >2013-03-09 12:30:34,828 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) [368aa39f] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 12:30:34,829 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) [368aa39f] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 12:30:34,829 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) [368aa39f] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 12:30:34,830 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) [368aa39f] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 12:30:34,830 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) [368aa39f] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 12:30:34,830 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) [368aa39f] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 12:30:34,830 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) [368aa39f] Attempting to update VM OVFs in Data Center Default >2013-03-09 12:30:34,831 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) [368aa39f] Successfully updated VM OVFs in Data Center Default >2013-03-09 12:30:34,831 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) [368aa39f] Attempting to update template OVFs in Data Center Default >2013-03-09 12:30:34,832 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) [368aa39f] Succesfully updated templates OVFs in Data Center Default >2013-03-09 12:30:34,832 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) [368aa39f] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 12:30:34,832 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-28) [368aa39f] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 13:30:33,789 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-1) [7228cbd3] Start refreshing all users data >2013-03-09 13:30:34,832 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-38) [5d712191] Attempting to update VMs/Templates Ovf. >2013-03-09 13:30:34,833 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-38) [5d712191] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 13:30:34,834 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-38) [5d712191] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 13:30:34,834 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-38) [5d712191] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 13:30:34,835 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-38) [5d712191] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 13:30:34,835 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-38) [5d712191] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 13:30:34,836 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-38) [5d712191] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 13:30:34,836 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-38) [5d712191] Attempting to update VM OVFs in Data Center Default >2013-03-09 13:30:34,837 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-38) [5d712191] Successfully updated VM OVFs in Data Center Default >2013-03-09 13:30:34,837 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-38) [5d712191] Attempting to update template OVFs in Data Center Default >2013-03-09 13:30:34,837 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-38) [5d712191] Succesfully updated templates OVFs in Data Center Default >2013-03-09 13:30:34,837 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-38) [5d712191] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 13:30:34,838 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-38) [5d712191] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 14:30:33,791 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-33) [68a361a0] Start refreshing all users data >2013-03-09 14:30:34,838 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-97) [309c8303] Attempting to update VMs/Templates Ovf. >2013-03-09 14:30:34,839 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-97) [309c8303] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 14:30:34,839 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-97) [309c8303] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 14:30:34,839 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-97) [309c8303] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 14:30:34,840 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-97) [309c8303] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 14:30:34,840 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-97) [309c8303] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 14:30:34,841 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-97) [309c8303] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 14:30:34,841 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-97) [309c8303] Attempting to update VM OVFs in Data Center Default >2013-03-09 14:30:34,842 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-97) [309c8303] Successfully updated VM OVFs in Data Center Default >2013-03-09 14:30:34,842 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-97) [309c8303] Attempting to update template OVFs in Data Center Default >2013-03-09 14:30:34,842 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-97) [309c8303] Succesfully updated templates OVFs in Data Center Default >2013-03-09 14:30:34,842 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-97) [309c8303] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 14:30:34,843 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-97) [309c8303] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 15:30:33,793 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-46) [58449771] Start refreshing all users data >2013-03-09 15:30:34,843 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [dd0195b] Attempting to update VMs/Templates Ovf. >2013-03-09 15:30:34,844 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [dd0195b] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 15:30:34,845 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [dd0195b] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 15:30:34,845 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [dd0195b] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 15:30:34,846 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [dd0195b] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 15:30:34,846 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [dd0195b] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 15:30:34,846 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [dd0195b] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 15:30:34,846 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [dd0195b] Attempting to update VM OVFs in Data Center Default >2013-03-09 15:30:34,847 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [dd0195b] Successfully updated VM OVFs in Data Center Default >2013-03-09 15:30:34,847 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [dd0195b] Attempting to update template OVFs in Data Center Default >2013-03-09 15:30:34,848 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [dd0195b] Succesfully updated templates OVFs in Data Center Default >2013-03-09 15:30:34,848 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [dd0195b] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 15:30:34,848 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [dd0195b] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 16:30:33,795 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-22) [59973c55] Start refreshing all users data >2013-03-09 16:30:34,848 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [903e742] Attempting to update VMs/Templates Ovf. >2013-03-09 16:30:34,849 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [903e742] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 16:30:34,850 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [903e742] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 16:30:34,850 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [903e742] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 16:30:34,850 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [903e742] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 16:30:34,850 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [903e742] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 16:30:34,851 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [903e742] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 16:30:34,851 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [903e742] Attempting to update VM OVFs in Data Center Default >2013-03-09 16:30:34,852 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [903e742] Successfully updated VM OVFs in Data Center Default >2013-03-09 16:30:34,852 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [903e742] Attempting to update template OVFs in Data Center Default >2013-03-09 16:30:34,852 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [903e742] Succesfully updated templates OVFs in Data Center Default >2013-03-09 16:30:34,852 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [903e742] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 16:30:34,853 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [903e742] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 17:30:33,797 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-91) [1ce1a6db] Start refreshing all users data >2013-03-09 17:30:34,853 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-68) [1902f6b0] Attempting to update VMs/Templates Ovf. >2013-03-09 17:30:34,854 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-68) [1902f6b0] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 17:30:34,855 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-68) [1902f6b0] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 17:30:34,855 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-68) [1902f6b0] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 17:30:34,856 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-68) [1902f6b0] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 17:30:34,856 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-68) [1902f6b0] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 17:30:34,856 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-68) [1902f6b0] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 17:30:34,856 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-68) [1902f6b0] Attempting to update VM OVFs in Data Center Default >2013-03-09 17:30:34,857 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-68) [1902f6b0] Successfully updated VM OVFs in Data Center Default >2013-03-09 17:30:34,857 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-68) [1902f6b0] Attempting to update template OVFs in Data Center Default >2013-03-09 17:30:34,858 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-68) [1902f6b0] Succesfully updated templates OVFs in Data Center Default >2013-03-09 17:30:34,858 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-68) [1902f6b0] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 17:30:34,858 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-68) [1902f6b0] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 18:30:33,799 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-89) [34cfa5f2] Start refreshing all users data >2013-03-09 18:30:34,858 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [3d58d304] Attempting to update VMs/Templates Ovf. >2013-03-09 18:30:34,859 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [3d58d304] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 18:30:34,860 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [3d58d304] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 18:30:34,860 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [3d58d304] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 18:30:34,860 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [3d58d304] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 18:30:34,860 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [3d58d304] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 18:30:34,861 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [3d58d304] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 18:30:34,861 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [3d58d304] Attempting to update VM OVFs in Data Center Default >2013-03-09 18:30:34,862 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [3d58d304] Successfully updated VM OVFs in Data Center Default >2013-03-09 18:30:34,862 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [3d58d304] Attempting to update template OVFs in Data Center Default >2013-03-09 18:30:34,862 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [3d58d304] Succesfully updated templates OVFs in Data Center Default >2013-03-09 18:30:34,862 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [3d58d304] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 18:30:34,863 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [3d58d304] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 19:30:33,800 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-97) [309c8303] Start refreshing all users data >2013-03-09 19:30:34,863 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-3) [e986c15] Attempting to update VMs/Templates Ovf. >2013-03-09 19:30:34,864 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-3) [e986c15] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 19:30:34,865 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-3) [e986c15] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 19:30:34,865 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-3) [e986c15] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 19:30:34,866 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-3) [e986c15] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 19:30:34,866 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-3) [e986c15] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 19:30:34,866 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-3) [e986c15] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 19:30:34,866 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-3) [e986c15] Attempting to update VM OVFs in Data Center Default >2013-03-09 19:30:34,867 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-3) [e986c15] Successfully updated VM OVFs in Data Center Default >2013-03-09 19:30:34,867 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-3) [e986c15] Attempting to update template OVFs in Data Center Default >2013-03-09 19:30:34,868 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-3) [e986c15] Succesfully updated templates OVFs in Data Center Default >2013-03-09 19:30:34,868 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-3) [e986c15] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 19:30:34,868 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-3) [e986c15] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 20:30:33,802 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-19) [a9105b1] Start refreshing all users data >2013-03-09 20:30:34,868 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [1f3593ef] Attempting to update VMs/Templates Ovf. >2013-03-09 20:30:34,869 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [1f3593ef] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 20:30:34,870 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [1f3593ef] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 20:30:34,870 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [1f3593ef] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 20:30:34,871 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [1f3593ef] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 20:30:34,871 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [1f3593ef] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 20:30:34,871 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [1f3593ef] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 20:30:34,871 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [1f3593ef] Attempting to update VM OVFs in Data Center Default >2013-03-09 20:30:34,874 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [1f3593ef] Successfully updated VM OVFs in Data Center Default >2013-03-09 20:30:34,874 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [1f3593ef] Attempting to update template OVFs in Data Center Default >2013-03-09 20:30:34,874 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [1f3593ef] Succesfully updated templates OVFs in Data Center Default >2013-03-09 20:30:34,874 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [1f3593ef] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 20:30:34,875 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [1f3593ef] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 21:30:33,805 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-90) [727ae14e] Start refreshing all users data >2013-03-09 21:30:34,875 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-87) [5a31881] Attempting to update VMs/Templates Ovf. >2013-03-09 21:30:34,876 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-87) [5a31881] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 21:30:34,883 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-87) [5a31881] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 21:30:34,883 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-87) [5a31881] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 21:30:34,886 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-87) [5a31881] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 21:30:34,886 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-87) [5a31881] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 21:30:34,887 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-87) [5a31881] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 21:30:34,887 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-87) [5a31881] Attempting to update VM OVFs in Data Center Default >2013-03-09 21:30:34,887 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-87) [5a31881] Successfully updated VM OVFs in Data Center Default >2013-03-09 21:30:34,887 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-87) [5a31881] Attempting to update template OVFs in Data Center Default >2013-03-09 21:30:34,888 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-87) [5a31881] Succesfully updated templates OVFs in Data Center Default >2013-03-09 21:30:34,888 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-87) [5a31881] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 21:30:34,889 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-87) [5a31881] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 22:30:33,807 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-8) [70a3eb0a] Start refreshing all users data >2013-03-09 22:30:34,889 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [7685c42e] Attempting to update VMs/Templates Ovf. >2013-03-09 22:30:34,890 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [7685c42e] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 22:30:34,891 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [7685c42e] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 22:30:34,891 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [7685c42e] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 22:30:34,892 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [7685c42e] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 22:30:34,892 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [7685c42e] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 22:30:34,893 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [7685c42e] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 22:30:34,893 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [7685c42e] Attempting to update VM OVFs in Data Center Default >2013-03-09 22:30:34,894 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [7685c42e] Successfully updated VM OVFs in Data Center Default >2013-03-09 22:30:34,894 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [7685c42e] Attempting to update template OVFs in Data Center Default >2013-03-09 22:30:34,894 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [7685c42e] Succesfully updated templates OVFs in Data Center Default >2013-03-09 22:30:34,894 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [7685c42e] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 22:30:34,895 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [7685c42e] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-09 23:30:33,808 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-82) [68b8a2e0] Start refreshing all users data >2013-03-09 23:30:34,895 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) [49ebe838] Attempting to update VMs/Templates Ovf. >2013-03-09 23:30:34,896 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) [49ebe838] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 23:30:34,897 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) [49ebe838] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 23:30:34,897 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) [49ebe838] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 23:30:34,898 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) [49ebe838] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 23:30:34,898 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) [49ebe838] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 23:30:34,898 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) [49ebe838] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-09 23:30:34,899 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) [49ebe838] Attempting to update VM OVFs in Data Center Default >2013-03-09 23:30:34,899 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) [49ebe838] Successfully updated VM OVFs in Data Center Default >2013-03-09 23:30:34,899 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) [49ebe838] Attempting to update template OVFs in Data Center Default >2013-03-09 23:30:34,900 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) [49ebe838] Succesfully updated templates OVFs in Data Center Default >2013-03-09 23:30:34,900 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) [49ebe838] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-09 23:30:34,901 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-100) [49ebe838] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-10 00:30:33,810 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-24) [3113df82] Start refreshing all users data >2013-03-10 00:30:34,901 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [2aee83e2] Attempting to update VMs/Templates Ovf. >2013-03-10 00:30:34,902 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [2aee83e2] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 00:30:34,903 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [2aee83e2] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 00:30:34,903 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [2aee83e2] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 00:30:34,904 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [2aee83e2] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 00:30:34,904 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [2aee83e2] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 00:30:34,904 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [2aee83e2] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 00:30:34,904 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [2aee83e2] Attempting to update VM OVFs in Data Center Default >2013-03-10 00:30:34,905 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [2aee83e2] Successfully updated VM OVFs in Data Center Default >2013-03-10 00:30:34,905 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [2aee83e2] Attempting to update template OVFs in Data Center Default >2013-03-10 00:30:34,906 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [2aee83e2] Succesfully updated templates OVFs in Data Center Default >2013-03-10 00:30:34,906 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [2aee83e2] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-10 00:30:34,906 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [2aee83e2] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-10 01:30:33,812 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-54) [1d8e03e4] Start refreshing all users data >2013-03-10 01:30:34,906 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [6399daff] Attempting to update VMs/Templates Ovf. >2013-03-10 01:30:34,907 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [6399daff] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 01:30:34,908 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [6399daff] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 01:30:34,908 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [6399daff] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 01:30:34,909 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [6399daff] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 01:30:34,909 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [6399daff] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 01:30:34,910 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [6399daff] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 01:30:34,910 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [6399daff] Attempting to update VM OVFs in Data Center Default >2013-03-10 01:30:34,910 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [6399daff] Successfully updated VM OVFs in Data Center Default >2013-03-10 01:30:34,910 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [6399daff] Attempting to update template OVFs in Data Center Default >2013-03-10 01:30:34,911 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [6399daff] Succesfully updated templates OVFs in Data Center Default >2013-03-10 01:30:34,911 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [6399daff] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-10 01:30:34,912 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [6399daff] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-10 02:30:33,815 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-34) [626594b9] Start refreshing all users data >2013-03-10 02:30:34,912 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-57) [7161258c] Attempting to update VMs/Templates Ovf. >2013-03-10 02:30:34,914 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-57) [7161258c] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 02:30:34,921 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-57) [7161258c] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 02:30:34,921 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-57) [7161258c] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 02:30:34,923 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-57) [7161258c] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 02:30:34,923 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-57) [7161258c] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 02:30:34,924 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-57) [7161258c] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 02:30:34,924 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-57) [7161258c] Attempting to update VM OVFs in Data Center Default >2013-03-10 02:30:34,925 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-57) [7161258c] Successfully updated VM OVFs in Data Center Default >2013-03-10 02:30:34,925 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-57) [7161258c] Attempting to update template OVFs in Data Center Default >2013-03-10 02:30:34,926 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-57) [7161258c] Succesfully updated templates OVFs in Data Center Default >2013-03-10 02:30:34,926 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-57) [7161258c] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-10 02:30:34,927 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-57) [7161258c] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-10 03:30:33,817 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-36) [16379607] Start refreshing all users data >2013-03-10 03:30:34,928 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) [757b1f7d] Attempting to update VMs/Templates Ovf. >2013-03-10 03:30:34,929 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) [757b1f7d] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 03:30:34,929 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) [757b1f7d] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 03:30:34,929 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) [757b1f7d] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 03:30:34,930 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) [757b1f7d] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 03:30:34,930 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) [757b1f7d] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 03:30:34,931 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) [757b1f7d] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 03:30:34,931 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) [757b1f7d] Attempting to update VM OVFs in Data Center Default >2013-03-10 03:30:34,931 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) [757b1f7d] Successfully updated VM OVFs in Data Center Default >2013-03-10 03:30:34,931 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) [757b1f7d] Attempting to update template OVFs in Data Center Default >2013-03-10 03:30:34,932 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) [757b1f7d] Succesfully updated templates OVFs in Data Center Default >2013-03-10 03:30:34,932 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) [757b1f7d] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-10 03:30:34,933 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-47) [757b1f7d] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-10 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (QuartzScheduler_Worker-63) [7819a10c] AuditLogCleanupManager::deleteAgedOutAuditLogs - entered >2013-03-10 04:30:33,819 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-47) [757b1f7d] Start refreshing all users data >2013-03-10 04:30:34,933 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-65) [617179c6] Attempting to update VMs/Templates Ovf. >2013-03-10 04:30:34,934 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-65) [617179c6] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 04:30:34,935 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-65) [617179c6] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 04:30:34,935 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-65) [617179c6] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 04:30:34,936 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-65) [617179c6] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 04:30:34,936 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-65) [617179c6] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 04:30:34,936 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-65) [617179c6] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 04:30:34,936 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-65) [617179c6] Attempting to update VM OVFs in Data Center Default >2013-03-10 04:30:34,937 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-65) [617179c6] Successfully updated VM OVFs in Data Center Default >2013-03-10 04:30:34,937 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-65) [617179c6] Attempting to update template OVFs in Data Center Default >2013-03-10 04:30:34,938 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-65) [617179c6] Succesfully updated templates OVFs in Data Center Default >2013-03-10 04:30:34,938 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-65) [617179c6] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-10 04:30:34,939 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-65) [617179c6] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-10 05:30:33,822 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-73) [10e3e76a] Start refreshing all users data >2013-03-10 05:30:34,939 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-23) [1b1b0943] Attempting to update VMs/Templates Ovf. >2013-03-10 05:30:34,940 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-23) [1b1b0943] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 05:30:34,941 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-23) [1b1b0943] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 05:30:34,941 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-23) [1b1b0943] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 05:30:34,941 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-23) [1b1b0943] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 05:30:34,941 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-23) [1b1b0943] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 05:30:34,942 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-23) [1b1b0943] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 05:30:34,942 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-23) [1b1b0943] Attempting to update VM OVFs in Data Center Default >2013-03-10 05:30:34,943 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-23) [1b1b0943] Successfully updated VM OVFs in Data Center Default >2013-03-10 05:30:34,943 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-23) [1b1b0943] Attempting to update template OVFs in Data Center Default >2013-03-10 05:30:34,943 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-23) [1b1b0943] Succesfully updated templates OVFs in Data Center Default >2013-03-10 05:30:34,943 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-23) [1b1b0943] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-10 05:30:34,944 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-23) [1b1b0943] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-10 06:30:33,824 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-74) [7ff19b05] Start refreshing all users data >2013-03-10 06:30:34,944 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Attempting to update VMs/Templates Ovf. >2013-03-10 06:30:34,945 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 06:30:34,946 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 06:30:34,946 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 06:30:34,947 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 06:30:34,947 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 06:30:34,948 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 06:30:34,948 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Attempting to update VM OVFs in Data Center Default >2013-03-10 06:30:34,948 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Successfully updated VM OVFs in Data Center Default >2013-03-10 06:30:34,949 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Attempting to update template OVFs in Data Center Default >2013-03-10 06:30:34,949 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Succesfully updated templates OVFs in Data Center Default >2013-03-10 06:30:34,949 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-10 06:30:34,950 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [cf9e7f6] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-10 07:30:33,826 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-21) [54a87db] Start refreshing all users data >2013-03-10 07:30:34,950 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-9) [1799d88f] Attempting to update VMs/Templates Ovf. >2013-03-10 07:30:34,951 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-9) [1799d88f] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 07:30:34,952 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-9) [1799d88f] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 07:30:34,952 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-9) [1799d88f] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 07:30:34,953 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-9) [1799d88f] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 07:30:34,953 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-9) [1799d88f] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 07:30:34,953 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-9) [1799d88f] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 07:30:34,953 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-9) [1799d88f] Attempting to update VM OVFs in Data Center Default >2013-03-10 07:30:34,954 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-9) [1799d88f] Successfully updated VM OVFs in Data Center Default >2013-03-10 07:30:34,954 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-9) [1799d88f] Attempting to update template OVFs in Data Center Default >2013-03-10 07:30:34,955 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-9) [1799d88f] Succesfully updated templates OVFs in Data Center Default >2013-03-10 07:30:34,955 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-9) [1799d88f] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-10 07:30:34,955 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-9) [1799d88f] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-10 08:30:33,828 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-32) [248f5a32] Start refreshing all users data >2013-03-10 08:30:34,956 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [f64b7f4] Attempting to update VMs/Templates Ovf. >2013-03-10 08:30:34,957 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [f64b7f4] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 08:30:34,958 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [f64b7f4] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 08:30:34,958 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [f64b7f4] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 08:30:34,958 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [f64b7f4] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 08:30:34,958 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [f64b7f4] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 08:30:34,959 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [f64b7f4] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 08:30:34,959 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [f64b7f4] Attempting to update VM OVFs in Data Center Default >2013-03-10 08:30:34,960 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [f64b7f4] Successfully updated VM OVFs in Data Center Default >2013-03-10 08:30:34,960 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [f64b7f4] Attempting to update template OVFs in Data Center Default >2013-03-10 08:30:34,960 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [f64b7f4] Succesfully updated templates OVFs in Data Center Default >2013-03-10 08:30:34,960 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [f64b7f4] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-10 08:30:34,961 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-70) [f64b7f4] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-10 09:30:33,831 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-70) [f64b7f4] Start refreshing all users data >2013-03-10 09:30:34,962 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) [1e82ea3b] Attempting to update VMs/Templates Ovf. >2013-03-10 09:30:34,963 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) [1e82ea3b] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 09:30:34,963 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) [1e82ea3b] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 09:30:34,963 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) [1e82ea3b] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 09:30:34,964 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) [1e82ea3b] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 09:30:34,964 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) [1e82ea3b] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 09:30:34,965 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) [1e82ea3b] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 09:30:34,965 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) [1e82ea3b] Attempting to update VM OVFs in Data Center Default >2013-03-10 09:30:34,965 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) [1e82ea3b] Successfully updated VM OVFs in Data Center Default >2013-03-10 09:30:34,966 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) [1e82ea3b] Attempting to update template OVFs in Data Center Default >2013-03-10 09:30:34,966 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) [1e82ea3b] Succesfully updated templates OVFs in Data Center Default >2013-03-10 09:30:34,966 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) [1e82ea3b] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-10 09:30:34,967 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-64) [1e82ea3b] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-10 10:30:33,833 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-16) [7161c0df] Start refreshing all users data >2013-03-10 10:30:34,967 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Attempting to update VMs/Templates Ovf. >2013-03-10 10:30:34,968 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 10:30:34,969 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 10:30:34,969 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 10:30:34,969 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 10:30:34,969 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 10:30:34,970 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 10:30:34,970 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Attempting to update VM OVFs in Data Center Default >2013-03-10 10:30:34,971 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Successfully updated VM OVFs in Data Center Default >2013-03-10 10:30:34,971 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Attempting to update template OVFs in Data Center Default >2013-03-10 10:30:34,972 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Succesfully updated templates OVFs in Data Center Default >2013-03-10 10:30:34,972 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Attempting to remove uneeded template/vm OVFs in Data Center Default >2013-03-10 10:30:34,972 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Succesfully removed unneeded template/vm OVFs in Data Center Default >2013-03-10 10:46:01,708 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) domain 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126:D1 in problem. vds: cheetah02.scl.lab.tlv.redhat.com >2013-03-10 10:46:35,041 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-67) [2d511931] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-10 10:46:38,049 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-21) [54a87db] ResourceManager::refreshVdsRunTimeInfo::Failed to refresh VDS , vds = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2013-03-10 10:46:39,957 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (QuartzScheduler_Worker-6) [36698780] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 10:46:39,972 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) [36698780] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2013-03-10 10:46:41,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [7ff19b05] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7381db48 >2013-03-10 10:46:41,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [7ff19b05] FINISH, GetHardwareInfoVDSCommand, log id: 7381db48 >2013-03-10 10:46:41,224 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-74) [4e8523d] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 10:46:41,228 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-74) [432a8e4d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 10:46:41,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-74) [432a8e4d] START, ConnectStorageServerVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, storageType = NFS, connectionList = [{ id: 4e470cfe-b114-4fb0-8f52-fc538d2b740b, connection: lion.qa.lab.tlv.redhat.com:/Fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 491c7287 >2013-03-10 10:46:41,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (QuartzScheduler_Worker-74) [432a8e4d] FINISH, ConnectStorageServerVDSCommand, return: {4e470cfe-b114-4fb0-8f52-fc538d2b740b=0}, log id: 491c7287 >2013-03-10 10:46:41,264 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-74) [432a8e4d] Host cheetah02.scl.lab.tlv.redhat.com storage connection was succeeded >2013-03-10 10:46:41,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 497fa14a >2013-03-10 10:47:48,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 10:47:48,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 10:47:48,519 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 10:47:48,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 497fa14a >2013-03-10 10:47:48,524 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [2df5b8c6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 10:47:48,538 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (pool-3-thread-46) [2df5b8c6] START, ResetIrsVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, vdsId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, ignoreStopFailed = true), log id: 23e8d7af >2013-03-10 10:47:48,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-3-thread-46) [2df5b8c6] START, SpmStopVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3), log id: f8a6231 >2013-03-10 10:47:48,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-3-thread-46) [2df5b8c6] SpmStopVDSCommand:: vds cheetah02.scl.lab.tlv.redhat.com is in Connecting status - not performing spm stop, pool id 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 10:47:48,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-3-thread-46) [2df5b8c6] FINISH, SpmStopVDSCommand, log id: f8a6231 >2013-03-10 10:47:48,542 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (pool-3-thread-46) [2df5b8c6] FINISH, ResetIrsVDSCommand, log id: 23e8d7af >2013-03-10 10:47:48,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [2df5b8c6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 58f9c058 >2013-03-10 10:47:48,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [2df5b8c6] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 10:47:48,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [2df5b8c6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 58f9c058 >2013-03-10 10:47:48,550 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [432a8e4d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 10:47:48,562 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-74) [608eb8ef] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 10:47:48,563 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [608eb8ef] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5e6572c0 >2013-03-10 10:47:48,564 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [608eb8ef] FINISH, SetVdsStatusVDSCommand, log id: 5e6572c0 >2013-03-10 10:47:48,572 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-74) [608eb8ef] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 10:47:48,595 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-74) [641760de] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 10:47:48,598 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-74) [75083f49] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 10:47:48,599 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-74) [75083f49] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 10:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [765ac3f2] Autorecovering 1 hosts >2013-03-10 10:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [765ac3f2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 10:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [414742d6] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 10:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [414742d6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 10:50:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [414742d6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 46ba38c3 >2013-03-10 10:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [414742d6] FINISH, SetVdsStatusVDSCommand, log id: 46ba38c3 >2013-03-10 10:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-13) [414742d6] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 42758195 >2013-03-10 10:50:00,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [414742d6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5016445c >2013-03-10 10:50:00,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [414742d6] FINISH, GetHardwareInfoVDSCommand, log id: 5016445c >2013-03-10 10:50:00,277 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-13) [87fc7eb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 10:50:00,280 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-13) [6c137223] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 10:50:00,281 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-13) [6c137223] FINISH, ActivateVdsVDSCommand, log id: 42758195 >2013-03-10 10:50:00,304 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [6c137223] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 10:50:00,308 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [6c137223] Autorecovering 1 storage domains >2013-03-10 10:50:00,308 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [6c137223] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 10:50:00,309 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [9dc7ab] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 10:50:00,309 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [9dc7ab] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 10:50 AM >2013-03-10 10:50:00,315 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [9dc7ab] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 10:50 AM >2013-03-10 10:50:00,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [5e26a1e0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 51fb8073 >2013-03-10 10:50:00,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [5e26a1e0] FINISH, GetHardwareInfoVDSCommand, log id: 51fb8073 >2013-03-10 10:50:00,783 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-48) [3bebbf68] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 10:50:00,787 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-48) [3b30f53f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 10:50:00,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1e61692c >2013-03-10 10:51:03,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 10:51:03,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 10:51:03,743 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 10:51:03,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 1e61692c >2013-03-10 10:51:03,747 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [242d5433] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 10:51:03,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [242d5433] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4f1443d >2013-03-10 10:51:03,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [242d5433] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 10:51:03,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [242d5433] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4f1443d >2013-03-10 10:51:03,765 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [242d5433] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 10:51:03,765 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [3b30f53f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 10:51:03,765 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [3b30f53f] The following operation Event [eventType=DOMAINFAILOVER, storagePoolId=5849b030-626e-47cb-ad90-3ce782d831b3, domainId=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, vdsId=null] was cancelled, because of recosntruct was run before >2013-03-10 10:51:03,777 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-48) [530a3e2c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 10:51:03,779 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [530a3e2c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 69cad111 >2013-03-10 10:51:03,780 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [530a3e2c] FINISH, SetVdsStatusVDSCommand, log id: 69cad111 >2013-03-10 10:51:03,785 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-48) [530a3e2c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 10:51:03,807 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-48) [29771e12] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 10:51:03,810 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-48) [77e00969] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 10:51:03,811 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-48) [77e00969] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 10:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [7e7f6470] Autorecovering 1 hosts >2013-03-10 10:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [7e7f6470] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 10:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [789d455d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 10:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [789d455d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 10:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [789d455d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 121e8f66 >2013-03-10 10:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [789d455d] FINISH, SetVdsStatusVDSCommand, log id: 121e8f66 >2013-03-10 10:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-27) [789d455d] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 278fbb1e >2013-03-10 10:55:00,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [789d455d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3f60c939 >2013-03-10 10:55:00,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [789d455d] FINISH, GetHardwareInfoVDSCommand, log id: 3f60c939 >2013-03-10 10:55:00,421 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-27) [5f9846a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 10:55:00,424 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-27) [db2aee8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 10:55:00,425 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-27) [db2aee8] FINISH, ActivateVdsVDSCommand, log id: 278fbb1e >2013-03-10 10:55:00,445 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [db2aee8] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 10:55:00,446 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-27) [db2aee8] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 10:55:00,447 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [db2aee8] Autorecovering 1 storage domains >2013-03-10 10:55:00,447 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [db2aee8] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 10:55:00,448 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [3f8e56c4] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 10:55:00,448 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [3f8e56c4] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 10:55 AM >2013-03-10 10:55:00,451 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [3f8e56c4] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 10:55 AM >2013-03-10 10:55:00,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [68b8a2e0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5bcc45d9 >2013-03-10 10:55:01,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [68b8a2e0] FINISH, GetHardwareInfoVDSCommand, log id: 5bcc45d9 >2013-03-10 10:55:01,052 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-82) [7f8fd9ca] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 10:55:01,056 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-82) [142ca429] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 10:55:01,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1b5798b0 >2013-03-10 10:56:04,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 10:56:04,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 10:56:04,087 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 10:56:04,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 1b5798b0 >2013-03-10 10:56:04,091 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [25f1ed8a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 10:56:04,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [25f1ed8a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 43f13cc1 >2013-03-10 10:56:04,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [25f1ed8a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 10:56:04,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [25f1ed8a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 43f13cc1 >2013-03-10 10:56:04,108 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [25f1ed8a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 10:56:04,108 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [142ca429] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 10:56:04,118 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-82) [797f8406] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 10:56:04,120 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [797f8406] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 53728a36 >2013-03-10 10:56:04,121 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [797f8406] FINISH, SetVdsStatusVDSCommand, log id: 53728a36 >2013-03-10 10:56:04,127 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-82) [797f8406] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 10:56:04,148 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-82) [59bdd615] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 10:56:04,150 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-82) [49ff21ac] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 10:56:04,151 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-82) [49ff21ac] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 11:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [1a249fa6] Autorecovering 1 hosts >2013-03-10 11:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [1a249fa6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [bf6211b] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [bf6211b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:00:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [bf6211b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3e7b7e38 >2013-03-10 11:00:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [bf6211b] FINISH, SetVdsStatusVDSCommand, log id: 3e7b7e38 >2013-03-10 11:00:00,019 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-83) [bf6211b] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 384eda5d >2013-03-10 11:00:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [bf6211b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1e5b60ae >2013-03-10 11:00:00,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [bf6211b] FINISH, GetHardwareInfoVDSCommand, log id: 1e5b60ae >2013-03-10 11:00:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-83) [38fc3f29] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:00:00,206 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-83) [6c8bb907] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:00:00,207 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-83) [6c8bb907] FINISH, ActivateVdsVDSCommand, log id: 384eda5d >2013-03-10 11:00:00,230 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [6c8bb907] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:00:00,231 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-83) [6c8bb907] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 11:00:00,232 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [6c8bb907] Autorecovering 1 storage domains >2013-03-10 11:00:00,232 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [6c8bb907] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 11:00:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [76a4f65b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:00:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [76a4f65b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:00 AM >2013-03-10 11:00:00,239 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [76a4f65b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:00 AM >2013-03-10 11:00:01,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [6facedac] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3554f766 >2013-03-10 11:00:01,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [6facedac] FINISH, GetHardwareInfoVDSCommand, log id: 3554f766 >2013-03-10 11:00:01,390 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-75) [64a517e] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 11:00:01,393 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-75) [4209e32c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 11:00:01,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7309356e >2013-03-10 11:01:04,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 11:01:04,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:01:04,325 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 11:01:04,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 7309356e >2013-03-10 11:01:04,329 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [17f1e53d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:01:04,343 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [17f1e53d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 48672526 >2013-03-10 11:01:04,343 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [17f1e53d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 11:01:04,343 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [17f1e53d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 48672526 >2013-03-10 11:01:04,346 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [17f1e53d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 11:01:04,346 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [4209e32c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 11:01:04,356 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-75) [155e8972] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:01:04,357 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [155e8972] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6ab85573 >2013-03-10 11:01:04,359 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [155e8972] FINISH, SetVdsStatusVDSCommand, log id: 6ab85573 >2013-03-10 11:01:04,364 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-75) [155e8972] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 11:01:04,393 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [6af49176] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:01:04,395 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [4b3a2fbf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:01:04,397 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-75) [4b3a2fbf] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 11:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [3d58d304] Autorecovering 1 hosts >2013-03-10 11:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [3d58d304] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [682fff90] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [682fff90] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [682fff90] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4bea5865 >2013-03-10 11:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [682fff90] FINISH, SetVdsStatusVDSCommand, log id: 4bea5865 >2013-03-10 11:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-2) [682fff90] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 47139762 >2013-03-10 11:05:00,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [682fff90] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 69f5e619 >2013-03-10 11:05:00,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [682fff90] FINISH, GetHardwareInfoVDSCommand, log id: 69f5e619 >2013-03-10 11:05:00,408 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [18c097ac] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:05:00,410 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [43d8833b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:05:00,411 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-2) [43d8833b] FINISH, ActivateVdsVDSCommand, log id: 47139762 >2013-03-10 11:05:00,431 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [43d8833b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:05:00,432 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [43d8833b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 11:05:00,433 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [43d8833b] Autorecovering 1 storage domains >2013-03-10 11:05:00,433 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [43d8833b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 11:05:00,434 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [43d6587e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:05:00,434 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [43d6587e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:05 AM >2013-03-10 11:05:00,437 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [43d6587e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:05 AM >2013-03-10 11:05:01,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [5a31881] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: e76acc9 >2013-03-10 11:05:01,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [5a31881] FINISH, GetHardwareInfoVDSCommand, log id: e76acc9 >2013-03-10 11:05:01,637 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-87) [770807e0] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 11:05:01,640 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-87) [4f95575c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 11:05:01,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 439f98b3 >2013-03-10 11:06:04,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 11:06:04,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:06:04,564 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 11:06:04,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 439f98b3 >2013-03-10 11:06:04,567 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [26827b8c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:06:04,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [26827b8c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3924800a >2013-03-10 11:06:04,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [26827b8c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 11:06:04,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [26827b8c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3924800a >2013-03-10 11:06:04,583 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [26827b8c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 11:06:04,583 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [4f95575c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 11:06:04,594 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-87) [74e860ed] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:06:04,595 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [74e860ed] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1b6aa422 >2013-03-10 11:06:04,596 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [74e860ed] FINISH, SetVdsStatusVDSCommand, log id: 1b6aa422 >2013-03-10 11:06:04,603 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [74e860ed] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 11:06:04,627 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [155d657b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:06:04,629 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [7d84b609] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:06:04,631 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-87) [7d84b609] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 11:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [1e4d912e] Autorecovering 1 hosts >2013-03-10 11:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [1e4d912e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [6aac0eed] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [6aac0eed] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [6aac0eed] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 401b3ea8 >2013-03-10 11:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [6aac0eed] FINISH, SetVdsStatusVDSCommand, log id: 401b3ea8 >2013-03-10 11:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [6aac0eed] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 688182b5 >2013-03-10 11:10:00,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [6aac0eed] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 705ba9c2 >2013-03-10 11:10:00,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [6aac0eed] FINISH, GetHardwareInfoVDSCommand, log id: 705ba9c2 >2013-03-10 11:10:00,188 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [4a0fec6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:10:00,191 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [3864e41d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:10:00,194 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [3864e41d] FINISH, ActivateVdsVDSCommand, log id: 688182b5 >2013-03-10 11:10:00,214 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [3864e41d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:10:00,215 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-94) [3864e41d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 11:10:00,216 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [3864e41d] Autorecovering 1 storage domains >2013-03-10 11:10:00,216 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [3864e41d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 11:10:00,217 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [3f6684f9] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:10:00,217 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [3f6684f9] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:10 AM >2013-03-10 11:10:00,220 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [3f6684f9] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:10 AM >2013-03-10 11:10:01,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [1e82ea3b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: f2ab702 >2013-03-10 11:10:01,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [1e82ea3b] FINISH, GetHardwareInfoVDSCommand, log id: f2ab702 >2013-03-10 11:10:01,876 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-64) [4245e0e4] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 11:10:01,880 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-64) [6ca84608] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 11:10:01,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 24f8a8ce >2013-03-10 11:11:04,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 11:11:04,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:11:04,805 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 11:11:04,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 24f8a8ce >2013-03-10 11:11:04,809 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [29a21eec] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:11:04,823 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [29a21eec] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5bbd383c >2013-03-10 11:11:04,823 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [29a21eec] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 11:11:04,823 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [29a21eec] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5bbd383c >2013-03-10 11:11:04,825 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [29a21eec] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 11:11:04,826 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [6ca84608] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 11:11:04,835 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-64) [5339cbe] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:11:04,837 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [5339cbe] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4cad8308 >2013-03-10 11:11:04,838 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [5339cbe] FINISH, SetVdsStatusVDSCommand, log id: 4cad8308 >2013-03-10 11:11:04,844 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-64) [5339cbe] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 11:11:04,865 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-64) [26a0548e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:11:04,868 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-64) [1ccdcf1a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:11:04,869 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-64) [1ccdcf1a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 11:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [727ae14e] Autorecovering 1 hosts >2013-03-10 11:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [727ae14e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [5265c71a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [5265c71a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [5265c71a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5e5d77a7 >2013-03-10 11:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [5265c71a] FINISH, SetVdsStatusVDSCommand, log id: 5e5d77a7 >2013-03-10 11:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-90) [5265c71a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 78584764 >2013-03-10 11:15:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [5265c71a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5320d2a9 >2013-03-10 11:15:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [5265c71a] FINISH, GetHardwareInfoVDSCommand, log id: 5320d2a9 >2013-03-10 11:15:00,258 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-90) [150cdae7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:15:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-90) [64b9d153] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:15:00,262 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-90) [64b9d153] FINISH, ActivateVdsVDSCommand, log id: 78584764 >2013-03-10 11:15:00,281 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [64b9d153] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:15:00,282 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [64b9d153] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 11:15:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [64b9d153] Autorecovering 1 storage domains >2013-03-10 11:15:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [64b9d153] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 11:15:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [2ff1908c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:15:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [2ff1908c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:15 AM >2013-03-10 11:15:00,287 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [2ff1908c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:15 AM >2013-03-10 11:15:02,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [dfb5ca5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 456a3245 >2013-03-10 11:15:02,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [dfb5ca5] FINISH, GetHardwareInfoVDSCommand, log id: 456a3245 >2013-03-10 11:15:02,113 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-78) [57b57845] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 11:15:02,117 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-78) [41bb7ce6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 11:15:02,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5d1509e8 >2013-03-10 11:16:05,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 11:16:05,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:16:05,008 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 11:16:05,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 5d1509e8 >2013-03-10 11:16:05,012 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [6cbf374e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:16:05,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6cbf374e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 78c39974 >2013-03-10 11:16:05,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [6cbf374e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 11:16:05,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6cbf374e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 78c39974 >2013-03-10 11:16:05,028 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [6cbf374e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 11:16:05,028 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [41bb7ce6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 11:16:05,041 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-78) [2137aea] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:16:05,042 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [2137aea] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3afabe8c >2013-03-10 11:16:05,044 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [2137aea] FINISH, SetVdsStatusVDSCommand, log id: 3afabe8c >2013-03-10 11:16:05,049 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-78) [2137aea] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 11:16:05,069 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-78) [44779431] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:16:05,072 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-78) [5081fd3d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:16:05,073 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-78) [5081fd3d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 11:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [6da5caa3] Autorecovering 1 hosts >2013-03-10 11:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [6da5caa3] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [1f26b68e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [1f26b68e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [1f26b68e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4a528d39 >2013-03-10 11:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [1f26b68e] FINISH, SetVdsStatusVDSCommand, log id: 4a528d39 >2013-03-10 11:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [1f26b68e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 23fc80a >2013-03-10 11:20:00,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [1f26b68e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: d7164fd >2013-03-10 11:20:00,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [1f26b68e] FINISH, GetHardwareInfoVDSCommand, log id: d7164fd >2013-03-10 11:20:00,188 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [20b7cbec] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:20:00,191 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [3119ebaf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:20:00,192 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [3119ebaf] FINISH, ActivateVdsVDSCommand, log id: 23fc80a >2013-03-10 11:20:00,211 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [3119ebaf] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:20:00,212 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [3119ebaf] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 11:20:00,213 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [3119ebaf] Autorecovering 1 storage domains >2013-03-10 11:20:00,213 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [3119ebaf] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 11:20:00,214 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [1c455426] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:20:00,214 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [1c455426] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:20 AM >2013-03-10 11:20:00,217 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [1c455426] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:20 AM >2013-03-10 11:20:02,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [f64b7f4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2655f722 >2013-03-10 11:20:02,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [f64b7f4] FINISH, GetHardwareInfoVDSCommand, log id: 2655f722 >2013-03-10 11:20:02,319 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-70) [6cc99f82] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 11:20:02,323 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-70) [13f87896] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 11:20:02,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 441d6c9b >2013-03-10 11:21:06,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 11:21:06,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:21:06,305 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 11:21:06,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 441d6c9b >2013-03-10 11:21:06,309 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [280a2ce9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:21:06,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [280a2ce9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 191bf2ce >2013-03-10 11:21:06,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [280a2ce9] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 11:21:06,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [280a2ce9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 191bf2ce >2013-03-10 11:21:06,326 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [280a2ce9] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 11:21:06,326 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [13f87896] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 11:21:06,336 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-70) [5e2d7bd4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:21:06,337 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [5e2d7bd4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 113a810b >2013-03-10 11:21:06,338 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [5e2d7bd4] FINISH, SetVdsStatusVDSCommand, log id: 113a810b >2013-03-10 11:21:06,344 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-70) [5e2d7bd4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 11:21:06,375 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-70) [1998a7f8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:21:06,378 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-70) [5c93cf3e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:21:06,379 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-70) [5c93cf3e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 11:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-37) [6a5a04e] Autorecovering 1 hosts >2013-03-10 11:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-37) [6a5a04e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-37) [182e8f92] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-37) [182e8f92] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:25:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [182e8f92] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 78c63bc0 >2013-03-10 11:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [182e8f92] FINISH, SetVdsStatusVDSCommand, log id: 78c63bc0 >2013-03-10 11:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-37) [182e8f92] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6e8a372a >2013-03-10 11:25:00,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [182e8f92] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1e6c28fe >2013-03-10 11:25:00,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [182e8f92] FINISH, GetHardwareInfoVDSCommand, log id: 1e6c28fe >2013-03-10 11:25:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-37) [29adb613] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:25:00,208 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-37) [3d9653c8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:25:00,209 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-37) [3d9653c8] FINISH, ActivateVdsVDSCommand, log id: 6e8a372a >2013-03-10 11:25:00,228 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-37) [3d9653c8] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:25:00,229 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-37) [3d9653c8] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 11:25:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-37) [3d9653c8] Autorecovering 1 storage domains >2013-03-10 11:25:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-37) [3d9653c8] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 11:25:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-37) [305534eb] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:25:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-37) [305534eb] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:25 AM >2013-03-10 11:25:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-37) [305534eb] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:25 AM >2013-03-10 11:25:00,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [7819a10c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 19d01051 >2013-03-10 11:25:00,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [7819a10c] FINISH, GetHardwareInfoVDSCommand, log id: 19d01051 >2013-03-10 11:25:00,622 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-63) [71ae1aa4] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 11:25:00,626 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-63) [72f6147f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 11:25:00,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2a6bc319 >2013-03-10 11:26:03,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 11:26:03,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:26:03,623 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 11:26:03,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2a6bc319 >2013-03-10 11:26:03,626 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [932ad05] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:26:03,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [932ad05] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3d310546 >2013-03-10 11:26:03,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [932ad05] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 11:26:03,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [932ad05] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3d310546 >2013-03-10 11:26:03,642 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [932ad05] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 11:26:03,642 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [72f6147f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 11:26:03,652 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-63) [2c32441c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:26:03,654 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [2c32441c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 462f7da6 >2013-03-10 11:26:03,655 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [2c32441c] FINISH, SetVdsStatusVDSCommand, log id: 462f7da6 >2013-03-10 11:26:03,661 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-63) [2c32441c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 11:26:03,685 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-63) [63d347b9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:26:03,687 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-63) [23074657] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:26:03,689 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-63) [23074657] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 11:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [54a87db] Autorecovering 1 hosts >2013-03-10 11:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [54a87db] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [3218721b] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [3218721b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [3218721b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2eef00e4 >2013-03-10 11:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [3218721b] FINISH, SetVdsStatusVDSCommand, log id: 2eef00e4 >2013-03-10 11:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-21) [3218721b] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 58ea94bf >2013-03-10 11:30:00,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [3218721b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 41b8ade1 >2013-03-10 11:30:00,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [3218721b] FINISH, GetHardwareInfoVDSCommand, log id: 41b8ade1 >2013-03-10 11:30:00,206 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-21) [520b3e75] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:30:00,208 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-21) [79afe194] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:30:00,209 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-21) [79afe194] FINISH, ActivateVdsVDSCommand, log id: 58ea94bf >2013-03-10 11:30:00,229 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [79afe194] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:30:00,230 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-21) [79afe194] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 11:30:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [79afe194] Autorecovering 1 storage domains >2013-03-10 11:30:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [79afe194] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 11:30:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [6fde538c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:30:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [6fde538c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:30 AM >2013-03-10 11:30:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [6fde538c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:30 AM >2013-03-10 11:30:00,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [2ff6ef51] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4c427ff >2013-03-10 11:30:00,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [2ff6ef51] FINISH, GetHardwareInfoVDSCommand, log id: 4c427ff >2013-03-10 11:30:00,930 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-81) [5b92b545] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 11:30:00,933 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-81) [69ecae22] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 11:30:00,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3f92710d >2013-03-10 11:30:33,833 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-82) [49ff21ac] Start refreshing all users data >2013-03-10 11:30:34,973 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-30) [4e5c16e] Attempting to update VMs/Templates Ovf. >2013-03-10 11:30:34,974 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-30) [4e5c16e] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 11:30:34,975 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-30) [4e5c16e] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 11:30:34,975 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-30) [4e5c16e] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 11:30:34,975 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-30) [4e5c16e] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 11:30:34,975 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-30) [4e5c16e] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 11:30:34,976 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-30) [4e5c16e] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 11:31:03,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 11:31:03,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:31:03,878 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 11:31:03,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 3f92710d >2013-03-10 11:31:03,882 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [6f813b48] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:31:03,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6f813b48] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1b442d1b >2013-03-10 11:31:03,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [6f813b48] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 11:31:03,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6f813b48] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1b442d1b >2013-03-10 11:31:03,899 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [6f813b48] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 11:31:03,899 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [69ecae22] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 11:31:03,909 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-81) [73a9b60f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:31:03,911 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [73a9b60f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1c1cb9b8 >2013-03-10 11:31:03,912 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [73a9b60f] FINISH, SetVdsStatusVDSCommand, log id: 1c1cb9b8 >2013-03-10 11:31:03,917 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-81) [73a9b60f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 11:31:03,953 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-81) [64c61ded] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:31:03,955 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-81) [368a7c36] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:31:03,957 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-81) [368a7c36] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 11:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) [309c8303] Autorecovering 1 hosts >2013-03-10 11:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) [309c8303] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-97) [347a147e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-97) [347a147e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [347a147e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7a32c9f7 >2013-03-10 11:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [347a147e] FINISH, SetVdsStatusVDSCommand, log id: 7a32c9f7 >2013-03-10 11:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-97) [347a147e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6dff9455 >2013-03-10 11:35:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [347a147e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 57765bb6 >2013-03-10 11:35:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [347a147e] FINISH, GetHardwareInfoVDSCommand, log id: 57765bb6 >2013-03-10 11:35:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-97) [1e27833f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:35:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-97) [93f67a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:35:00,203 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-97) [93f67a] FINISH, ActivateVdsVDSCommand, log id: 6dff9455 >2013-03-10 11:35:00,222 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-97) [93f67a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:35:00,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-97) [93f67a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 11:35:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) [93f67a] Autorecovering 1 storage domains >2013-03-10 11:35:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) [93f67a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 11:35:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-97) [1a3ead3] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:35:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-97) [1a3ead3] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:35 AM >2013-03-10 11:35:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-97) [1a3ead3] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:35 AM >2013-03-10 11:35:01,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [305534eb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1780a3bb >2013-03-10 11:35:01,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [305534eb] FINISH, GetHardwareInfoVDSCommand, log id: 1780a3bb >2013-03-10 11:35:01,203 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-37) [2c3ae2c7] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 11:35:01,206 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-37) [301689eb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 11:35:01,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 78fe9e25 >2013-03-10 11:36:04,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 11:36:04,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:36:04,121 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 11:36:04,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 78fe9e25 >2013-03-10 11:36:04,125 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [72262d4a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:36:04,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [72262d4a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 379a3f13 >2013-03-10 11:36:04,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [72262d4a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 11:36:04,139 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [72262d4a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 379a3f13 >2013-03-10 11:36:04,141 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [72262d4a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 11:36:04,141 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [301689eb] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 11:36:04,151 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-37) [27ff7555] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:36:04,153 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [27ff7555] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 558083c2 >2013-03-10 11:36:04,154 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [27ff7555] FINISH, SetVdsStatusVDSCommand, log id: 558083c2 >2013-03-10 11:36:04,160 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-37) [27ff7555] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 11:36:04,181 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-37) [24789b67] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:36:04,184 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-37) [a2d4d2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:36:04,185 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-37) [a2d4d2] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 11:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [10e3e76a] Autorecovering 1 hosts >2013-03-10 11:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [10e3e76a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [1689ae4] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [1689ae4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [1689ae4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: e3d80cc >2013-03-10 11:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [1689ae4] FINISH, SetVdsStatusVDSCommand, log id: e3d80cc >2013-03-10 11:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [1689ae4] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 38a192a5 >2013-03-10 11:40:00,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [1689ae4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5c8fd89 >2013-03-10 11:40:00,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [1689ae4] FINISH, GetHardwareInfoVDSCommand, log id: 5c8fd89 >2013-03-10 11:40:00,188 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-73) [79f01f60] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:40:00,191 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-73) [68cd8347] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:40:00,192 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [68cd8347] FINISH, ActivateVdsVDSCommand, log id: 38a192a5 >2013-03-10 11:40:00,211 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [68cd8347] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:40:00,212 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-73) [68cd8347] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 11:40:00,213 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [68cd8347] Autorecovering 1 storage domains >2013-03-10 11:40:00,213 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [68cd8347] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 11:40:00,214 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [4ac01c82] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:40:00,214 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [4ac01c82] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:40 AM >2013-03-10 11:40:00,217 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [4ac01c82] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:40 AM >2013-03-10 11:40:01,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [6fde538c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 258d3ff >2013-03-10 11:40:01,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [6fde538c] FINISH, GetHardwareInfoVDSCommand, log id: 258d3ff >2013-03-10 11:40:01,428 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-21) [3b56a655] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 11:40:01,431 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-21) [6dc1b1b4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 11:40:01,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1a0ccead >2013-03-10 11:41:06,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 11:41:06,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:41:06,434 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 11:41:06,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 1a0ccead >2013-03-10 11:41:06,438 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [3631b852] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:41:06,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [3631b852] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4ace764a >2013-03-10 11:41:06,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [3631b852] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 11:41:06,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [3631b852] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4ace764a >2013-03-10 11:41:06,454 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [3631b852] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 11:41:06,454 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [6dc1b1b4] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 11:41:06,464 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-21) [598eb568] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:41:06,465 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [598eb568] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 377f97fa >2013-03-10 11:41:06,467 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [598eb568] FINISH, SetVdsStatusVDSCommand, log id: 377f97fa >2013-03-10 11:41:06,473 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-21) [598eb568] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 11:41:06,494 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-21) [17cc036] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:41:06,496 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-21) [44ec9806] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:41:06,497 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-21) [44ec9806] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 11:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [59973c55] Autorecovering 1 hosts >2013-03-10 11:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [59973c55] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [25929f87] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [25929f87] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [25929f87] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6a491c22 >2013-03-10 11:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [25929f87] FINISH, SetVdsStatusVDSCommand, log id: 6a491c22 >2013-03-10 11:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-22) [25929f87] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 54d8c725 >2013-03-10 11:45:00,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [25929f87] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 52a5ad45 >2013-03-10 11:45:00,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [25929f87] FINISH, GetHardwareInfoVDSCommand, log id: 52a5ad45 >2013-03-10 11:45:00,187 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [5a6b540e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:45:00,189 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [4bee76d2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:45:00,191 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-22) [4bee76d2] FINISH, ActivateVdsVDSCommand, log id: 54d8c725 >2013-03-10 11:45:00,210 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [4bee76d2] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:45:00,211 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [4bee76d2] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 11:45:00,212 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [4bee76d2] Autorecovering 1 storage domains >2013-03-10 11:45:00,212 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [4bee76d2] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 11:45:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [5b31b1b8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:45:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [5b31b1b8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:45 AM >2013-03-10 11:45:00,216 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [5b31b1b8] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:45 AM >2013-03-10 11:45:00,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [6fd08d94] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5075807d >2013-03-10 11:45:00,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [6fd08d94] FINISH, GetHardwareInfoVDSCommand, log id: 5075807d >2013-03-10 11:45:00,727 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-51) [7883a884] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 11:45:00,731 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-51) [883cab2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 11:45:00,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7186daa7 >2013-03-10 11:46:05,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 11:46:05,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:46:05,029 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 11:46:05,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 7186daa7 >2013-03-10 11:46:05,032 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [79162706] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:46:05,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [79162706] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: dd31d7f >2013-03-10 11:46:05,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [79162706] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 11:46:05,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [79162706] FINISH, MarkPoolInReconstructModeVDSCommand, log id: dd31d7f >2013-03-10 11:46:05,048 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [79162706] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 11:46:05,048 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [883cab2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 11:46:05,058 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-51) [13e922bf] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:46:05,060 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [13e922bf] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7a6f2a12 >2013-03-10 11:46:05,061 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [13e922bf] FINISH, SetVdsStatusVDSCommand, log id: 7a6f2a12 >2013-03-10 11:46:05,067 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-51) [13e922bf] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 11:46:05,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-51) [72cead5b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:46:05,090 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-51) [1e3fe091] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:46:05,091 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-51) [1e3fe091] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 11:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) [7d17da86] Autorecovering 1 hosts >2013-03-10 11:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) [7d17da86] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-85) [332bdbf3] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-85) [332bdbf3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-85) [332bdbf3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5bf0272a >2013-03-10 11:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-85) [332bdbf3] FINISH, SetVdsStatusVDSCommand, log id: 5bf0272a >2013-03-10 11:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-85) [332bdbf3] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2359fd79 >2013-03-10 11:50:00,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-85) [332bdbf3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 637c7151 >2013-03-10 11:50:00,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-85) [332bdbf3] FINISH, GetHardwareInfoVDSCommand, log id: 637c7151 >2013-03-10 11:50:00,212 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-85) [127b28ad] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:50:00,215 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-85) [79388b7f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:50:00,216 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-85) [79388b7f] FINISH, ActivateVdsVDSCommand, log id: 2359fd79 >2013-03-10 11:50:00,235 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-85) [79388b7f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:50:00,236 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-85) [79388b7f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 11:50:00,237 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) [79388b7f] Autorecovering 1 storage domains >2013-03-10 11:50:00,237 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) [79388b7f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 11:50:00,238 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-85) [45887f55] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:50:00,238 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-85) [45887f55] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:50 AM >2013-03-10 11:50:00,241 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-85) [45887f55] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:50 AM >2013-03-10 11:50:02,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [37913a0f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3d9c6696 >2013-03-10 11:50:02,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [37913a0f] FINISH, GetHardwareInfoVDSCommand, log id: 3d9c6696 >2013-03-10 11:50:02,358 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-44) [234429b0] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 11:50:02,362 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-44) [74ba68f0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 11:50:02,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 291fd2f0 >2013-03-10 11:51:07,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 11:51:07,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:51:07,346 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 11:51:07,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 291fd2f0 >2013-03-10 11:51:07,350 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [501e7991] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:51:07,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [501e7991] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 676aab36 >2013-03-10 11:51:07,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [501e7991] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 11:51:07,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [501e7991] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 676aab36 >2013-03-10 11:51:07,373 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [74ba68f0] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 11:51:07,382 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-44) [8b03400] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:51:07,384 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [8b03400] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3fe46b0f >2013-03-10 11:51:07,385 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [8b03400] FINISH, SetVdsStatusVDSCommand, log id: 3fe46b0f >2013-03-10 11:51:07,391 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-44) [8b03400] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 11:51:07,412 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-44) [38891cb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:51:07,414 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-44) [34a132dd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:51:07,416 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-44) [34a132dd] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 11:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [71ee8c1f] Autorecovering 1 hosts >2013-03-10 11:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [71ee8c1f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-49) [9c2e356] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-49) [9c2e356] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [9c2e356] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 12d84098 >2013-03-10 11:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [9c2e356] FINISH, SetVdsStatusVDSCommand, log id: 12d84098 >2013-03-10 11:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-49) [9c2e356] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3b8dfa72 >2013-03-10 11:55:00,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [9c2e356] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2bb54d2b >2013-03-10 11:55:00,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [9c2e356] FINISH, GetHardwareInfoVDSCommand, log id: 2bb54d2b >2013-03-10 11:55:00,209 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-49) [6a283919] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:55:00,211 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-49) [11ac7516] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:55:00,212 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-49) [11ac7516] FINISH, ActivateVdsVDSCommand, log id: 3b8dfa72 >2013-03-10 11:55:00,232 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-49) [11ac7516] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 11:55:00,233 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-49) [11ac7516] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 11:55:00,234 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [11ac7516] Autorecovering 1 storage domains >2013-03-10 11:55:00,234 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [11ac7516] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 11:55:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-49) [337f8c98] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:55:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-49) [337f8c98] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:55 AM >2013-03-10 11:55:00,237 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-49) [337f8c98] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:55 AM >2013-03-10 11:55:01,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [597dc8bb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 12b913fd >2013-03-10 11:55:01,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [597dc8bb] FINISH, GetHardwareInfoVDSCommand, log id: 12b913fd >2013-03-10 11:55:01,663 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-52) [d36cfd9] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 11:55:01,667 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-52) [27dd6da1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 11:55:01,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6292af36 >2013-03-10 11:56:05,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 11:56:05,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 11:56:05,718 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 11:56:05,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 6292af36 >2013-03-10 11:56:05,722 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [c6ee301] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 11:56:05,736 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [c6ee301] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 11e0f38 >2013-03-10 11:56:05,736 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [c6ee301] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 11:56:05,736 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [c6ee301] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 11e0f38 >2013-03-10 11:56:05,738 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [c6ee301] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 11:56:05,738 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [27dd6da1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 11:56:05,749 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-52) [1d0ac71f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:56:05,750 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [1d0ac71f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 169c8dce >2013-03-10 11:56:05,751 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [1d0ac71f] FINISH, SetVdsStatusVDSCommand, log id: 169c8dce >2013-03-10 11:56:05,757 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-52) [1d0ac71f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 11:56:05,778 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [5aa5cf6f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:56:05,781 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [1c748011] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 11:56:05,782 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-52) [1c748011] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 12:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [3b862665] Autorecovering 1 hosts >2013-03-10 12:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [3b862665] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [6d5fbe07] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [6d5fbe07] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:00:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [6d5fbe07] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2326c50e >2013-03-10 12:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [6d5fbe07] FINISH, SetVdsStatusVDSCommand, log id: 2326c50e >2013-03-10 12:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [6d5fbe07] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1ceb4f4d >2013-03-10 12:00:00,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [6d5fbe07] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: e96973e >2013-03-10 12:00:00,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [6d5fbe07] FINISH, GetHardwareInfoVDSCommand, log id: e96973e >2013-03-10 12:00:00,261 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [5d1d3827] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:00:00,264 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [77b88837] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:00:00,265 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [77b88837] FINISH, ActivateVdsVDSCommand, log id: 1ceb4f4d >2013-03-10 12:00:00,285 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [77b88837] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:00:00,286 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [77b88837] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 12:00:00,287 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [77b88837] Autorecovering 1 storage domains >2013-03-10 12:00:00,287 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [77b88837] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 12:00:00,288 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [2f798a3f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:00:00,288 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [2f798a3f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 12:00 PM >2013-03-10 12:00:00,291 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [2f798a3f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 12:00 PM >2013-03-10 12:00:02,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [757b1f7d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 58b5ba4c >2013-03-10 12:00:02,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [757b1f7d] FINISH, GetHardwareInfoVDSCommand, log id: 58b5ba4c >2013-03-10 12:00:03,022 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-47) [55800d5f] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 12:00:03,026 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-47) [5d2b7adf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 12:00:03,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7044fdde >2013-03-10 12:01:09,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 12:01:09,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:01:09,016 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 12:01:09,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 7044fdde >2013-03-10 12:01:09,020 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [75919de8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:01:09,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [75919de8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3dd474e6 >2013-03-10 12:01:09,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [75919de8] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 12:01:09,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [75919de8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3dd474e6 >2013-03-10 12:01:09,036 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [75919de8] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 12:01:09,037 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [5d2b7adf] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 12:01:09,046 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-47) [39f82bf1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:01:09,048 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [39f82bf1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3b0532a9 >2013-03-10 12:01:09,049 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [39f82bf1] FINISH, SetVdsStatusVDSCommand, log id: 3b0532a9 >2013-03-10 12:01:09,055 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [39f82bf1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 12:01:09,077 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [6537de7c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:01:09,079 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [6634de09] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:01:09,080 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-47) [6634de09] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 12:05:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [42313555] Autorecovering 1 hosts >2013-03-10 12:05:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [42313555] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-99) [2b7e82ba] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-99) [2b7e82ba] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [2b7e82ba] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 43e6c345 >2013-03-10 12:05:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [2b7e82ba] FINISH, SetVdsStatusVDSCommand, log id: 43e6c345 >2013-03-10 12:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-99) [2b7e82ba] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7a2ca240 >2013-03-10 12:05:00,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [2b7e82ba] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3a1f7689 >2013-03-10 12:05:00,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [2b7e82ba] FINISH, GetHardwareInfoVDSCommand, log id: 3a1f7689 >2013-03-10 12:05:00,194 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-99) [41ceec5d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:05:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-99) [75805a55] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:05:00,198 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-99) [75805a55] FINISH, ActivateVdsVDSCommand, log id: 7a2ca240 >2013-03-10 12:05:00,217 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-99) [75805a55] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:05:00,218 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-99) [75805a55] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 12:05:00,219 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [75805a55] Autorecovering 1 storage domains >2013-03-10 12:05:00,219 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [75805a55] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 12:05:00,220 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-99) [7e0cc51a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:05:00,220 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-99) [7e0cc51a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 12:05 PM >2013-03-10 12:05:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-99) [7e0cc51a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 12:05 PM >2013-03-10 12:05:00,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [6ca6376] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 497b9b1f >2013-03-10 12:05:00,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [6ca6376] FINISH, GetHardwareInfoVDSCommand, log id: 497b9b1f >2013-03-10 12:05:00,330 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-15) [4b8f4152] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 12:05:00,334 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-15) [4a7543b6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 12:05:00,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 727df86a >2013-03-10 12:06:06,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 12:06:06,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:06:06,314 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 12:06:06,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 727df86a >2013-03-10 12:06:06,318 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [56a9222] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:06:06,340 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [56a9222] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3bfc69f7 >2013-03-10 12:06:06,340 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [56a9222] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 12:06:06,340 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [56a9222] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3bfc69f7 >2013-03-10 12:06:06,343 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [56a9222] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 12:06:06,343 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [4a7543b6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 12:06:06,353 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-15) [7d56c021] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:06:06,354 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [7d56c021] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4e0f9afb >2013-03-10 12:06:06,355 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [7d56c021] FINISH, SetVdsStatusVDSCommand, log id: 4e0f9afb >2013-03-10 12:06:06,361 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-15) [7d56c021] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 12:06:06,384 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-15) [349419b7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:06:06,386 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-15) [7efe441c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:06:06,387 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-15) [7efe441c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 12:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [1d8e03e4] Autorecovering 1 hosts >2013-03-10 12:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [1d8e03e4] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [3b854638] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [3b854638] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [3b854638] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 48b1f720 >2013-03-10 12:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [3b854638] FINISH, SetVdsStatusVDSCommand, log id: 48b1f720 >2013-03-10 12:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [3b854638] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 50951da5 >2013-03-10 12:10:00,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [3b854638] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: dc1327e >2013-03-10 12:10:00,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [3b854638] FINISH, GetHardwareInfoVDSCommand, log id: dc1327e >2013-03-10 12:10:00,429 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [759638c0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:10:00,431 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [52e42453] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:10:00,433 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [52e42453] FINISH, ActivateVdsVDSCommand, log id: 50951da5 >2013-03-10 12:10:00,452 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [52e42453] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:10:00,453 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [52e42453] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 12:10:00,454 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [52e42453] Autorecovering 1 storage domains >2013-03-10 12:10:00,454 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [52e42453] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 12:10:00,455 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [293391a1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:10:00,455 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [293391a1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 12:10 PM >2013-03-10 12:10:00,458 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [293391a1] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 12:10 PM >2013-03-10 12:10:00,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [4b3a2fbf] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4ae132ca >2013-03-10 12:10:00,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [4b3a2fbf] FINISH, GetHardwareInfoVDSCommand, log id: 4ae132ca >2013-03-10 12:10:00,633 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-75) [5efd86a3] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 12:10:00,637 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-75) [4cb53bc8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 12:10:00,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5d196a7d >2013-03-10 12:11:05,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 12:11:05,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:11:05,638 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 12:11:05,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 5d196a7d >2013-03-10 12:11:05,642 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [29f4bab] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:11:05,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [29f4bab] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 712dc606 >2013-03-10 12:11:05,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [29f4bab] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 12:11:05,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [29f4bab] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 712dc606 >2013-03-10 12:11:05,671 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [29f4bab] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 12:11:05,671 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [4cb53bc8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 12:11:05,681 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-75) [19fe9f81] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:11:05,683 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [19fe9f81] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 17abf95 >2013-03-10 12:11:05,684 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [19fe9f81] FINISH, SetVdsStatusVDSCommand, log id: 17abf95 >2013-03-10 12:11:05,689 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-75) [19fe9f81] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 12:11:05,711 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [2e7f9d15] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:11:05,714 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [31966c82] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:11:05,715 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-75) [31966c82] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 12:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [7d84b609] Autorecovering 1 hosts >2013-03-10 12:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [7d84b609] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:15:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [6a49a615] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:15:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [6a49a615] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [6a49a615] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 788ef91f >2013-03-10 12:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [6a49a615] FINISH, SetVdsStatusVDSCommand, log id: 788ef91f >2013-03-10 12:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-87) [6a49a615] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1dbe00b >2013-03-10 12:15:00,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [6a49a615] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 567a8b28 >2013-03-10 12:15:00,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [6a49a615] FINISH, GetHardwareInfoVDSCommand, log id: 567a8b28 >2013-03-10 12:15:00,401 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [51860fe2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:15:00,404 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [72c0d5e8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:15:00,405 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-87) [72c0d5e8] FINISH, ActivateVdsVDSCommand, log id: 1dbe00b >2013-03-10 12:15:00,424 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [72c0d5e8] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:15:00,425 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [72c0d5e8] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 12:15:00,426 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [72c0d5e8] Autorecovering 1 storage domains >2013-03-10 12:15:00,426 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [72c0d5e8] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 12:15:00,427 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [1ba1324f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:15:00,427 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [1ba1324f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 12:15 PM >2013-03-10 12:15:00,430 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [1ba1324f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 12:15 PM >2013-03-10 12:15:02,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [6edef405] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1b97344f >2013-03-10 12:15:02,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [6edef405] FINISH, GetHardwareInfoVDSCommand, log id: 1b97344f >2013-03-10 12:15:02,957 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-76) [2e8e68de] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 12:15:02,960 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-76) [d8ce996] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 12:15:02,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4506a4ba >2013-03-10 12:16:09,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 12:16:09,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:16:09,942 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 12:16:09,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 4506a4ba >2013-03-10 12:16:09,946 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [262017b6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:16:09,960 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [262017b6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 42982950 >2013-03-10 12:16:09,960 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [262017b6] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 12:16:09,960 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [262017b6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 42982950 >2013-03-10 12:16:09,962 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [262017b6] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 12:16:09,962 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [d8ce996] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 12:16:09,971 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-76) [516a0f44] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:16:09,973 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [516a0f44] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1a16ba96 >2013-03-10 12:16:09,974 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [516a0f44] FINISH, SetVdsStatusVDSCommand, log id: 1a16ba96 >2013-03-10 12:16:09,979 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-76) [516a0f44] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 12:16:09,999 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-76) [2d02e6c9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:16:10,002 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-76) [1d2fdfa2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:16:10,003 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-76) [1d2fdfa2] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 12:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [3113df82] Autorecovering 1 hosts >2013-03-10 12:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [3113df82] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [4240c3f9] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [4240c3f9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [4240c3f9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7f67d0c7 >2013-03-10 12:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [4240c3f9] FINISH, SetVdsStatusVDSCommand, log id: 7f67d0c7 >2013-03-10 12:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-24) [4240c3f9] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2b7d42c3 >2013-03-10 12:20:00,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [4240c3f9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 511e34fe >2013-03-10 12:20:00,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [4240c3f9] FINISH, GetHardwareInfoVDSCommand, log id: 511e34fe >2013-03-10 12:20:00,446 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-24) [702dfcfe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:20:00,448 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-24) [4aeda8f8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:20:00,450 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-24) [4aeda8f8] FINISH, ActivateVdsVDSCommand, log id: 2b7d42c3 >2013-03-10 12:20:00,469 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [4aeda8f8] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:20:00,470 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-24) [4aeda8f8] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 12:20:00,471 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [4aeda8f8] Autorecovering 1 storage domains >2013-03-10 12:20:00,471 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [4aeda8f8] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 12:20:00,472 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [3b0749de] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:20:00,472 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [3b0749de] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 12:20 PM >2013-03-10 12:20:00,475 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [3b0749de] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 12:20 PM >2013-03-10 12:20:01,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [49ebe838] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: dba1c48 >2013-03-10 12:20:01,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [49ebe838] FINISH, GetHardwareInfoVDSCommand, log id: dba1c48 >2013-03-10 12:20:01,243 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-100) [1d0c10d8] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 12:20:01,247 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-100) [22de5bab] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 12:20:01,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 13e71218 >2013-03-10 12:21:06,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 12:21:06,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:21:06,246 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 12:21:06,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 13e71218 >2013-03-10 12:21:06,250 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [7519f187] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:21:06,264 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [7519f187] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7ccfdda8 >2013-03-10 12:21:06,264 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [7519f187] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 12:21:06,264 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [7519f187] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7ccfdda8 >2013-03-10 12:21:06,266 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [7519f187] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 12:21:06,267 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [22de5bab] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 12:21:06,276 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-100) [7fea7a6d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:21:06,278 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [7fea7a6d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3863c36f >2013-03-10 12:21:06,279 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [7fea7a6d] FINISH, SetVdsStatusVDSCommand, log id: 3863c36f >2013-03-10 12:21:06,285 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-100) [7fea7a6d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 12:21:06,307 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-100) [536982b8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:21:06,309 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-100) [4edfbe75] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:21:06,310 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-100) [4edfbe75] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 12:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [7c0a6aba] Autorecovering 1 hosts >2013-03-10 12:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [7c0a6aba] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [4645fba5] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [4645fba5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [4645fba5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 16d1819d >2013-03-10 12:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [4645fba5] FINISH, SetVdsStatusVDSCommand, log id: 16d1819d >2013-03-10 12:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [4645fba5] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 44c2ecc7 >2013-03-10 12:25:00,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [4645fba5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 33485b96 >2013-03-10 12:25:00,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [4645fba5] FINISH, GetHardwareInfoVDSCommand, log id: 33485b96 >2013-03-10 12:25:00,329 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-84) [309e35b8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:25:00,331 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-84) [3f843087] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:25:00,333 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [3f843087] FINISH, ActivateVdsVDSCommand, log id: 44c2ecc7 >2013-03-10 12:25:00,352 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [3f843087] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:25:00,353 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-84) [3f843087] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 12:25:00,354 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [3f843087] Autorecovering 1 storage domains >2013-03-10 12:25:00,354 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [3f843087] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 12:25:00,355 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [664159d3] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:25:00,355 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [664159d3] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 12:25 PM >2013-03-10 12:25:00,358 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [664159d3] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 12:25 PM >2013-03-10 12:25:00,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [9dc7ab] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3029820e >2013-03-10 12:25:00,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [9dc7ab] FINISH, GetHardwareInfoVDSCommand, log id: 3029820e >2013-03-10 12:25:00,554 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-13) [60dcabc0] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 12:25:00,557 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-13) [7bd04b66] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 12:25:00,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 224febed >2013-03-10 12:26:04,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 12:26:04,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:26:04,593 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 12:26:04,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 224febed >2013-03-10 12:26:04,597 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [5b3e6555] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:26:04,611 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [5b3e6555] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7fd7f2ca >2013-03-10 12:26:04,611 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [5b3e6555] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 12:26:04,611 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [5b3e6555] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7fd7f2ca >2013-03-10 12:26:04,613 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [5b3e6555] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 12:26:04,614 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [7bd04b66] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 12:26:04,623 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-13) [6787bc2c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:26:04,625 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [6787bc2c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2779df45 >2013-03-10 12:26:04,626 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [6787bc2c] FINISH, SetVdsStatusVDSCommand, log id: 2779df45 >2013-03-10 12:26:04,631 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-13) [6787bc2c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 12:26:04,660 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-13) [3a0ef4a7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:26:04,663 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-13) [e31a467] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:26:04,664 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-13) [e31a467] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 12:30:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-81) [368a7c36] Autorecovering 1 hosts >2013-03-10 12:30:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-81) [368a7c36] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-81) [1d3aec4a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-81) [1d3aec4a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [1d3aec4a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6f06ed8e >2013-03-10 12:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [1d3aec4a] FINISH, SetVdsStatusVDSCommand, log id: 6f06ed8e >2013-03-10 12:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-81) [1d3aec4a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 677b05c5 >2013-03-10 12:30:00,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [1d3aec4a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7dbcc8cc >2013-03-10 12:30:00,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [1d3aec4a] FINISH, GetHardwareInfoVDSCommand, log id: 7dbcc8cc >2013-03-10 12:30:00,287 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-81) [521f3e69] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:30:00,289 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-81) [1da7e24b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:30:00,290 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-81) [1da7e24b] FINISH, ActivateVdsVDSCommand, log id: 677b05c5 >2013-03-10 12:30:00,310 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-81) [1da7e24b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:30:00,311 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-81) [1da7e24b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 12:30:00,312 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-81) [1da7e24b] Autorecovering 1 storage domains >2013-03-10 12:30:00,312 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-81) [1da7e24b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 12:30:00,313 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-81) [22c212ac] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:30:00,313 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-81) [22c212ac] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 12:30 PM >2013-03-10 12:30:00,315 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-81) [22c212ac] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 12:30 PM >2013-03-10 12:30:01,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [4afe4c39] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 173e104e >2013-03-10 12:30:01,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [4afe4c39] FINISH, GetHardwareInfoVDSCommand, log id: 173e104e >2013-03-10 12:30:01,915 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-69) [1003d323] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 12:30:01,919 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-69) [45a3abf9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 12:30:01,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 42c14525 >2013-03-10 12:30:33,835 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-79) [770cf360] Start refreshing all users data >2013-03-10 12:30:34,977 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) [4a66fd3e] Attempting to update VMs/Templates Ovf. >2013-03-10 12:30:34,978 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) [4a66fd3e] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 12:30:34,978 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) [4a66fd3e] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 12:30:34,978 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) [4a66fd3e] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 12:30:34,979 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) [4a66fd3e] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 12:30:34,979 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) [4a66fd3e] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 12:30:34,980 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-50) [4a66fd3e] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 12:31:06,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 12:31:06,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:31:06,922 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 12:31:06,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 42c14525 >2013-03-10 12:31:06,926 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [26918e3e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:31:06,939 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [26918e3e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7bc0afd4 >2013-03-10 12:31:06,939 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [26918e3e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 12:31:06,939 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [26918e3e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7bc0afd4 >2013-03-10 12:31:06,942 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [26918e3e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 12:31:06,942 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [45a3abf9] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 12:31:06,951 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-69) [428893e0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:31:06,953 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [428893e0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 39d84e9e >2013-03-10 12:31:06,954 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [428893e0] FINISH, SetVdsStatusVDSCommand, log id: 39d84e9e >2013-03-10 12:31:06,960 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-69) [428893e0] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 12:31:06,982 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-69) [fd995e2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:31:06,985 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-69) [58eb6d74] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:31:06,986 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-69) [58eb6d74] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 12:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [1b7b7a84] Autorecovering 1 hosts >2013-03-10 12:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [1b7b7a84] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [2a709587] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [2a709587] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [2a709587] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 59775b87 >2013-03-10 12:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [2a709587] FINISH, SetVdsStatusVDSCommand, log id: 59775b87 >2013-03-10 12:35:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [2a709587] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 44434d25 >2013-03-10 12:35:00,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [2a709587] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6cc100b1 >2013-03-10 12:35:00,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [2a709587] FINISH, GetHardwareInfoVDSCommand, log id: 6cc100b1 >2013-03-10 12:35:00,374 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-42) [6009a31a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:35:00,376 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-42) [5167d67e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:35:00,377 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [5167d67e] FINISH, ActivateVdsVDSCommand, log id: 44434d25 >2013-03-10 12:35:00,396 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [5167d67e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:35:00,397 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-42) [5167d67e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 12:35:00,399 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [5167d67e] Autorecovering 1 storage domains >2013-03-10 12:35:00,399 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [5167d67e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 12:35:00,399 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [7d6b044d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:35:00,399 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [7d6b044d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 12:35 PM >2013-03-10 12:35:00,402 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [7d6b044d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 12:35 PM >2013-03-10 12:35:01,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [50e78925] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 62de6400 >2013-03-10 12:35:01,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [50e78925] FINISH, GetHardwareInfoVDSCommand, log id: 62de6400 >2013-03-10 12:35:01,227 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-95) [491d8786] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 12:35:01,231 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-95) [61ec43ef] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 12:35:01,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1634aeee >2013-03-10 12:36:05,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 12:36:05,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:36:05,206 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 12:36:05,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 1634aeee >2013-03-10 12:36:05,210 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [4bbb8e7a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:36:05,224 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [4bbb8e7a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 29e18276 >2013-03-10 12:36:05,224 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [4bbb8e7a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 12:36:05,224 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [4bbb8e7a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 29e18276 >2013-03-10 12:36:05,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [4bbb8e7a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 12:36:05,227 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [61ec43ef] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 12:36:05,237 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-95) [13fb845] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:36:05,238 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [13fb845] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 35d397a2 >2013-03-10 12:36:05,240 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [13fb845] FINISH, SetVdsStatusVDSCommand, log id: 35d397a2 >2013-03-10 12:36:05,246 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-95) [13fb845] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 12:36:05,273 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-95) [21a1ed83] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:36:05,279 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-95) [2aa7b284] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:36:05,280 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-95) [2aa7b284] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 12:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [44ec9806] Autorecovering 1 hosts >2013-03-10 12:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [44ec9806] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [4f8516aa] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [4f8516aa] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [4f8516aa] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 63fcedd2 >2013-03-10 12:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [4f8516aa] FINISH, SetVdsStatusVDSCommand, log id: 63fcedd2 >2013-03-10 12:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-21) [4f8516aa] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2e5ec5fd >2013-03-10 12:40:00,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [4f8516aa] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6255ef81 >2013-03-10 12:40:00,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [4f8516aa] FINISH, GetHardwareInfoVDSCommand, log id: 6255ef81 >2013-03-10 12:40:00,636 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-21) [3c5e8466] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:40:00,638 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-21) [6450e509] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:40:00,639 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-21) [6450e509] FINISH, ActivateVdsVDSCommand, log id: 2e5ec5fd >2013-03-10 12:40:00,659 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [6450e509] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:40:00,660 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-21) [6450e509] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 12:40:00,661 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [6450e509] Autorecovering 1 storage domains >2013-03-10 12:40:00,661 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [6450e509] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 12:40:00,662 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [722b2686] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:40:00,662 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [722b2686] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 12:40 PM >2013-03-10 12:40:00,664 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [722b2686] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 12:40 PM >2013-03-10 12:40:02,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [22c212ac] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6b353f6 >2013-03-10 12:40:02,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [22c212ac] FINISH, GetHardwareInfoVDSCommand, log id: 6b353f6 >2013-03-10 12:40:02,552 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-81) [31f6d535] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 12:40:02,555 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-81) [5baca904] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 12:40:02,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 59b071b3 >2013-03-10 12:41:08,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 12:41:08,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:41:08,552 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 12:41:08,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 59b071b3 >2013-03-10 12:41:08,556 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [234095dc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:41:08,570 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [234095dc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c84c42f >2013-03-10 12:41:08,570 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [234095dc] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 12:41:08,570 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [234095dc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c84c42f >2013-03-10 12:41:08,572 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [234095dc] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 12:41:08,573 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [5baca904] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 12:41:08,583 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-81) [5d306095] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:41:08,584 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [5d306095] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 51c9cb0e >2013-03-10 12:41:08,585 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [5d306095] FINISH, SetVdsStatusVDSCommand, log id: 51c9cb0e >2013-03-10 12:41:08,591 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-81) [5d306095] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 12:41:08,612 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-81) [435d71b6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:41:08,615 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-81) [16c58dbf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:41:08,616 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-81) [16c58dbf] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 12:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [7161c0df] Autorecovering 1 hosts >2013-03-10 12:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [7161c0df] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [751187a7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [751187a7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [751187a7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 60cc31f7 >2013-03-10 12:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [751187a7] FINISH, SetVdsStatusVDSCommand, log id: 60cc31f7 >2013-03-10 12:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [751187a7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 126eff7e >2013-03-10 12:45:00,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [751187a7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 47220268 >2013-03-10 12:45:00,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [751187a7] FINISH, GetHardwareInfoVDSCommand, log id: 47220268 >2013-03-10 12:45:00,094 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-16) [21ac69b2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:45:00,097 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-16) [2ce6ca5d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:45:00,098 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [2ce6ca5d] FINISH, ActivateVdsVDSCommand, log id: 126eff7e >2013-03-10 12:45:00,117 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [2ce6ca5d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:45:00,118 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-16) [2ce6ca5d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 12:45:00,119 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [2ce6ca5d] Autorecovering 1 storage domains >2013-03-10 12:45:00,120 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [2ce6ca5d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 12:45:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [23a1f3ef] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:45:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [23a1f3ef] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 12:45 PM >2013-03-10 12:45:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [23a1f3ef] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 12:45 PM >2013-03-10 12:45:02,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [76a4f65b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 655ceed5 >2013-03-10 12:45:02,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [76a4f65b] FINISH, GetHardwareInfoVDSCommand, log id: 655ceed5 >2013-03-10 12:45:02,860 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-83) [19cfacd0] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 12:45:02,864 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-83) [43f4b6c5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 12:45:02,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 47697e6f >2013-03-10 12:46:09,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 12:46:09,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:46:09,466 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 12:46:09,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 47697e6f >2013-03-10 12:46:09,469 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [2c388885] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:46:09,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [2c388885] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 558752eb >2013-03-10 12:46:09,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [2c388885] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 12:46:09,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [2c388885] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 558752eb >2013-03-10 12:46:09,486 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [2c388885] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 12:46:09,486 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [43f4b6c5] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 12:46:09,497 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-83) [5fd5713e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:46:09,504 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [5fd5713e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3dc5423e >2013-03-10 12:46:09,509 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [5fd5713e] FINISH, SetVdsStatusVDSCommand, log id: 3dc5423e >2013-03-10 12:46:09,514 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-83) [5fd5713e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 12:46:09,544 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-83) [5e8f2a03] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:46:09,546 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-83) [1b38d5f2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:46:09,547 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-83) [1b38d5f2] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 12:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [6634de09] Autorecovering 1 hosts >2013-03-10 12:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [6634de09] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [39c1f26] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [39c1f26] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [39c1f26] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 474734b5 >2013-03-10 12:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [39c1f26] FINISH, SetVdsStatusVDSCommand, log id: 474734b5 >2013-03-10 12:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-47) [39c1f26] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4ac50d9c >2013-03-10 12:50:00,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [39c1f26] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4ab74f88 >2013-03-10 12:50:00,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [39c1f26] FINISH, GetHardwareInfoVDSCommand, log id: 4ab74f88 >2013-03-10 12:50:00,520 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [46b9b11e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:50:00,522 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [570a27f8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:50:00,524 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-47) [570a27f8] FINISH, ActivateVdsVDSCommand, log id: 4ac50d9c >2013-03-10 12:50:00,544 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [570a27f8] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:50:00,545 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [570a27f8] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 12:50:00,546 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [570a27f8] Autorecovering 1 storage domains >2013-03-10 12:50:00,546 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [570a27f8] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 12:50:00,547 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [56bc550] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:50:00,547 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [56bc550] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 12:50 PM >2013-03-10 12:50:00,549 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [56bc550] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 12:50 PM >2013-03-10 12:50:00,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [5c61f6bb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1c7e5972 >2013-03-10 12:50:00,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [5c61f6bb] FINISH, GetHardwareInfoVDSCommand, log id: 1c7e5972 >2013-03-10 12:50:00,790 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-86) [276b093b] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 12:50:00,794 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-86) [49942988] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 12:50:00,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3b5fd7da >2013-03-10 12:51:05,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 12:51:05,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:51:05,866 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 12:51:05,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 3b5fd7da >2013-03-10 12:51:05,870 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [169fe78c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:51:05,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [169fe78c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 67dbda09 >2013-03-10 12:51:05,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [169fe78c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 12:51:05,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [169fe78c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 67dbda09 >2013-03-10 12:51:05,891 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [169fe78c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 12:51:05,891 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [49942988] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 12:51:05,902 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-86) [3509fe95] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:51:05,903 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [3509fe95] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5977a568 >2013-03-10 12:51:05,905 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [3509fe95] FINISH, SetVdsStatusVDSCommand, log id: 5977a568 >2013-03-10 12:51:05,910 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-86) [3509fe95] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 12:51:05,932 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-86) [f4c5015] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:51:05,934 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-86) [4b4a3191] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:51:05,935 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-86) [4b4a3191] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 12:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [4a66fd3e] Autorecovering 1 hosts >2013-03-10 12:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [4a66fd3e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [4d486f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [4d486f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [4d486f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4c6d0085 >2013-03-10 12:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [4d486f] FINISH, SetVdsStatusVDSCommand, log id: 4c6d0085 >2013-03-10 12:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-50) [4d486f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7f544dbc >2013-03-10 12:55:00,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [4d486f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4b79fdf6 >2013-03-10 12:55:00,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [4d486f] FINISH, GetHardwareInfoVDSCommand, log id: 4b79fdf6 >2013-03-10 12:55:00,095 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-50) [79e6c081] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:55:00,097 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-50) [d3a53c0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:55:00,098 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-50) [d3a53c0] FINISH, ActivateVdsVDSCommand, log id: 7f544dbc >2013-03-10 12:55:00,118 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [d3a53c0] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 12:55:00,119 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-50) [d3a53c0] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 12:55:00,120 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [d3a53c0] Autorecovering 1 storage domains >2013-03-10 12:55:00,120 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [d3a53c0] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 12:55:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [66f48474] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:55:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [66f48474] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 12:55 PM >2013-03-10 12:55:00,124 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [66f48474] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 12:55 PM >2013-03-10 12:55:00,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [1c748011] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 37cfcde4 >2013-03-10 12:55:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [1c748011] FINISH, GetHardwareInfoVDSCommand, log id: 37cfcde4 >2013-03-10 12:55:00,197 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-52) [28b67905] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 12:55:00,200 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-52) [4f6e5a61] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 12:55:00,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: a7a74df >2013-03-10 12:56:04,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 12:56:04,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 12:56:04,565 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 12:56:04,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: a7a74df >2013-03-10 12:56:04,569 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [404284c0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 12:56:04,582 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [404284c0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 54be9b96 >2013-03-10 12:56:04,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [404284c0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 12:56:04,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [404284c0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 54be9b96 >2013-03-10 12:56:04,587 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [4f6e5a61] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 12:56:04,597 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-52) [2bc6f0f2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:56:04,599 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [2bc6f0f2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 57cab5ba >2013-03-10 12:56:04,600 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [2bc6f0f2] FINISH, SetVdsStatusVDSCommand, log id: 57cab5ba >2013-03-10 12:56:04,606 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-52) [2bc6f0f2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 12:56:04,627 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [151a06c9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:56:04,629 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [6cf8518] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 12:56:04,631 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-52) [6cf8518] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 13:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-66) [903e742] Autorecovering 1 hosts >2013-03-10 13:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-66) [903e742] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-66) [6d52ea3d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-66) [6d52ea3d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [6d52ea3d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 161200ea >2013-03-10 13:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [6d52ea3d] FINISH, SetVdsStatusVDSCommand, log id: 161200ea >2013-03-10 13:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-66) [6d52ea3d] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 24355be >2013-03-10 13:00:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [6d52ea3d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3d9d7ab3 >2013-03-10 13:00:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [6d52ea3d] FINISH, GetHardwareInfoVDSCommand, log id: 3d9d7ab3 >2013-03-10 13:00:00,258 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-66) [16381bf8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:00:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-66) [10ef07e1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:00:00,262 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-66) [10ef07e1] FINISH, ActivateVdsVDSCommand, log id: 24355be >2013-03-10 13:00:00,281 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-66) [10ef07e1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:00:00,282 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-66) [10ef07e1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 13:00:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-66) [10ef07e1] Autorecovering 1 storage domains >2013-03-10 13:00:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-66) [10ef07e1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 13:00:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-66) [40d50ad7] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:00:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-66) [40d50ad7] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 1:00 PM >2013-03-10 13:00:00,287 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-66) [40d50ad7] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 1:00 PM >2013-03-10 13:00:01,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [756a6735] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 113b9bef >2013-03-10 13:00:01,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [756a6735] FINISH, GetHardwareInfoVDSCommand, log id: 113b9bef >2013-03-10 13:00:01,876 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-88) [73a82305] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 13:00:01,880 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-88) [7da24f44] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 13:00:01,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7b123597 >2013-03-10 13:01:06,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 13:01:06,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:01:06,880 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 13:01:06,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 7b123597 >2013-03-10 13:01:06,884 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [367693e6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:01:06,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [367693e6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: f1c409a >2013-03-10 13:01:06,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [367693e6] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 13:01:06,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [367693e6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: f1c409a >2013-03-10 13:01:06,900 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [367693e6] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 13:01:06,901 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [7da24f44] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 13:01:06,910 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-88) [540a044c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:01:06,911 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [540a044c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 91b5697 >2013-03-10 13:01:06,913 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [540a044c] FINISH, SetVdsStatusVDSCommand, log id: 91b5697 >2013-03-10 13:01:06,918 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-88) [540a044c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 13:01:06,940 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-88) [7b0b17da] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:01:06,943 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-88) [6af286d9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:01:06,944 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-88) [6af286d9] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 13:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [617179c6] Autorecovering 1 hosts >2013-03-10 13:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [617179c6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [50a4bae1] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [50a4bae1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [50a4bae1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5fc1e860 >2013-03-10 13:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [50a4bae1] FINISH, SetVdsStatusVDSCommand, log id: 5fc1e860 >2013-03-10 13:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-65) [50a4bae1] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1f4d182b >2013-03-10 13:05:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [50a4bae1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 20a5008c >2013-03-10 13:05:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [50a4bae1] FINISH, GetHardwareInfoVDSCommand, log id: 20a5008c >2013-03-10 13:05:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-65) [558b863] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:05:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-65) [1e546b5f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:05:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-65) [1e546b5f] FINISH, ActivateVdsVDSCommand, log id: 1f4d182b >2013-03-10 13:05:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [1e546b5f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:05:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-65) [1e546b5f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 13:05:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [1e546b5f] Autorecovering 1 storage domains >2013-03-10 13:05:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [1e546b5f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 13:05:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [76a558c3] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:05:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [76a558c3] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 1:05 PM >2013-03-10 13:05:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [76a558c3] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 1:05 PM >2013-03-10 13:05:01,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-6) [36698780] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: b3ebe6a >2013-03-10 13:05:01,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-6) [36698780] FINISH, GetHardwareInfoVDSCommand, log id: b3ebe6a >2013-03-10 13:05:01,186 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-6) [2674b919] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 13:05:01,190 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-6) [40ae823d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 13:05:01,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6d8c673a >2013-03-10 13:06:05,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 13:06:05,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:06:05,505 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 13:06:05,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 6d8c673a >2013-03-10 13:06:05,508 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [60167f0f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:06:05,523 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [60167f0f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 17df02bb >2013-03-10 13:06:05,523 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [60167f0f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 13:06:05,523 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [60167f0f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 17df02bb >2013-03-10 13:06:05,525 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [60167f0f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 13:06:05,525 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [40ae823d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 13:06:05,536 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-6) [35d3efc4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:06:05,538 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-6) [35d3efc4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5ec4a27e >2013-03-10 13:06:05,539 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-6) [35d3efc4] FINISH, SetVdsStatusVDSCommand, log id: 5ec4a27e >2013-03-10 13:06:05,545 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-6) [35d3efc4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 13:06:05,567 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-6) [389bd458] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:06:05,569 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-6) [5e4b1f92] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:06:05,571 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-6) [5e4b1f92] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 13:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [6046fd1c] Autorecovering 1 hosts >2013-03-10 13:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [6046fd1c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-55) [274ea4f9] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-55) [274ea4f9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [274ea4f9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4829beb9 >2013-03-10 13:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [274ea4f9] FINISH, SetVdsStatusVDSCommand, log id: 4829beb9 >2013-03-10 13:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-55) [274ea4f9] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 46ec3029 >2013-03-10 13:10:00,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [274ea4f9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 26a34269 >2013-03-10 13:10:00,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [274ea4f9] FINISH, GetHardwareInfoVDSCommand, log id: 26a34269 >2013-03-10 13:10:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-55) [4ea7e2a4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:10:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-55) [753f12f6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:10:00,205 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-55) [753f12f6] FINISH, ActivateVdsVDSCommand, log id: 46ec3029 >2013-03-10 13:10:00,224 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-55) [753f12f6] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:10:00,225 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-55) [753f12f6] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 13:10:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [753f12f6] Autorecovering 1 storage domains >2013-03-10 13:10:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [753f12f6] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 13:10:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-55) [4fe38e0] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:10:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-55) [4fe38e0] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 1:10 PM >2013-03-10 13:10:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-55) [4fe38e0] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 1:10 PM >2013-03-10 13:10:02,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [5b31b1b8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2b3e2ae1 >2013-03-10 13:10:02,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [5b31b1b8] FINISH, GetHardwareInfoVDSCommand, log id: 2b3e2ae1 >2013-03-10 13:10:02,814 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-22) [178bde32] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 13:10:02,818 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-22) [e4f5410] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 13:10:02,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1fa00614 >2013-03-10 13:11:07,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 13:11:07,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:11:07,824 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 13:11:07,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 1fa00614 >2013-03-10 13:11:07,828 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [73edc54b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:11:07,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [73edc54b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 581aeb0a >2013-03-10 13:11:07,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [73edc54b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 13:11:07,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [73edc54b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 581aeb0a >2013-03-10 13:11:07,844 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [73edc54b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 13:11:07,844 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [e4f5410] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 13:11:07,854 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-22) [3ba3adf6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:11:07,855 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [3ba3adf6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7f0b5482 >2013-03-10 13:11:07,856 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [3ba3adf6] FINISH, SetVdsStatusVDSCommand, log id: 7f0b5482 >2013-03-10 13:11:07,862 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [3ba3adf6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 13:11:07,883 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [35f8fb5d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:11:07,885 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [331e8d6f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:11:07,887 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-22) [331e8d6f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 13:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [288cf14a] Autorecovering 1 hosts >2013-03-10 13:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [288cf14a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [42c958c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [42c958c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [42c958c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: e98d1ce >2013-03-10 13:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [42c958c] FINISH, SetVdsStatusVDSCommand, log id: e98d1ce >2013-03-10 13:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [42c958c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5526d701 >2013-03-10 13:15:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [42c958c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 18144b40 >2013-03-10 13:15:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [42c958c] FINISH, GetHardwareInfoVDSCommand, log id: 18144b40 >2013-03-10 13:15:00,258 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [4f99a76c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:15:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [163baf0d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:15:00,262 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [163baf0d] FINISH, ActivateVdsVDSCommand, log id: 5526d701 >2013-03-10 13:15:00,281 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [163baf0d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:15:00,282 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [163baf0d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 13:15:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [163baf0d] Autorecovering 1 storage domains >2013-03-10 13:15:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [163baf0d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 13:15:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [296259f6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:15:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [296259f6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 1:15 PM >2013-03-10 13:15:00,287 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [296259f6] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 1:15 PM >2013-03-10 13:15:02,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [2d511931] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 39f5dd4d >2013-03-10 13:15:02,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [2d511931] FINISH, GetHardwareInfoVDSCommand, log id: 39f5dd4d >2013-03-10 13:15:02,124 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-67) [7a842957] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 13:15:02,128 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-67) [9e751f0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 13:15:02,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 182abbb7 >2013-03-10 13:16:06,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 13:16:06,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:16:06,434 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 13:16:06,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 182abbb7 >2013-03-10 13:16:06,438 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [2541c79d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:16:06,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [2541c79d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7be8365a >2013-03-10 13:16:06,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [2541c79d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 13:16:06,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [2541c79d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7be8365a >2013-03-10 13:16:06,453 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [2541c79d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 13:16:06,454 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [9e751f0] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 13:16:06,463 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-67) [16c89cd3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:16:06,465 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [16c89cd3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5f9f15c1 >2013-03-10 13:16:06,466 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [16c89cd3] FINISH, SetVdsStatusVDSCommand, log id: 5f9f15c1 >2013-03-10 13:16:06,472 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-67) [16c89cd3] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 13:16:06,493 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-67) [2f988109] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:16:06,496 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-67) [3f6fdad6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:16:06,497 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-67) [3f6fdad6] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 13:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [58eb6d74] Autorecovering 1 hosts >2013-03-10 13:20:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [58eb6d74] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [365ef2fa] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [365ef2fa] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [365ef2fa] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 215ae667 >2013-03-10 13:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [365ef2fa] FINISH, SetVdsStatusVDSCommand, log id: 215ae667 >2013-03-10 13:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [365ef2fa] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4edc1627 >2013-03-10 13:20:00,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [365ef2fa] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3ed6a616 >2013-03-10 13:20:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [365ef2fa] FINISH, GetHardwareInfoVDSCommand, log id: 3ed6a616 >2013-03-10 13:20:00,257 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-69) [17684ff6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:20:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-69) [5d802840] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:20:00,261 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [5d802840] FINISH, ActivateVdsVDSCommand, log id: 4edc1627 >2013-03-10 13:20:00,280 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [5d802840] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:20:00,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-69) [5d802840] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 13:20:00,282 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [5d802840] Autorecovering 1 storage domains >2013-03-10 13:20:00,282 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [5d802840] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 13:20:00,283 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [4379195e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:20:00,283 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [4379195e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 1:20 PM >2013-03-10 13:20:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [4379195e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 1:20 PM >2013-03-10 13:20:00,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [4edfbe75] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1260d900 >2013-03-10 13:20:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [4edfbe75] FINISH, GetHardwareInfoVDSCommand, log id: 1260d900 >2013-03-10 13:20:00,736 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-100) [394fdb46] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 13:20:00,739 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-100) [639d2132] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 13:20:00,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5a5eada5 >2013-03-10 13:21:04,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 13:21:04,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:21:04,729 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 13:21:04,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 5a5eada5 >2013-03-10 13:21:04,736 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [261d2db9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:21:04,750 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [261d2db9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4f654052 >2013-03-10 13:21:04,750 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [261d2db9] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 13:21:04,750 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [261d2db9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4f654052 >2013-03-10 13:21:04,752 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [261d2db9] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 13:21:04,753 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [639d2132] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 13:21:04,765 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-100) [679ebedb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:21:04,766 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [679ebedb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 77269b69 >2013-03-10 13:21:04,767 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [679ebedb] FINISH, SetVdsStatusVDSCommand, log id: 77269b69 >2013-03-10 13:21:04,773 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-100) [679ebedb] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 13:21:04,800 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-100) [79217423] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:21:04,802 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-100) [316b7163] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:21:04,804 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-100) [316b7163] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 13:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [2aee83e2] Autorecovering 1 hosts >2013-03-10 13:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [2aee83e2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [442a47f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [442a47f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [442a47f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 19f1c192 >2013-03-10 13:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [442a47f] FINISH, SetVdsStatusVDSCommand, log id: 19f1c192 >2013-03-10 13:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [442a47f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 684b3134 >2013-03-10 13:25:00,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [442a47f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 19024746 >2013-03-10 13:25:00,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [442a47f] FINISH, GetHardwareInfoVDSCommand, log id: 19024746 >2013-03-10 13:25:00,410 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [5ff82eb4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:25:00,413 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [24d7654b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:25:00,414 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [24d7654b] FINISH, ActivateVdsVDSCommand, log id: 684b3134 >2013-03-10 13:25:00,433 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [24d7654b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:25:00,434 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [24d7654b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 13:25:00,435 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [24d7654b] Autorecovering 1 storage domains >2013-03-10 13:25:00,436 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [24d7654b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 13:25:00,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [347b8412] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:25:00,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [347b8412] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 1:25 PM >2013-03-10 13:25:00,439 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [347b8412] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 1:25 PM >2013-03-10 13:25:01,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [6399daff] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6b817a74 >2013-03-10 13:25:02,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [6399daff] FINISH, GetHardwareInfoVDSCommand, log id: 6b817a74 >2013-03-10 13:25:02,055 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-80) [38c16afe] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 13:25:02,059 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-80) [298e1ca7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 13:25:02,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7f3e9d43 >2013-03-10 13:26:06,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 13:26:06,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:26:06,110 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 13:26:06,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 7f3e9d43 >2013-03-10 13:26:06,114 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [6d8106f2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:26:06,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [6d8106f2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2674ff8c >2013-03-10 13:26:06,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [6d8106f2] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 13:26:06,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [6d8106f2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2674ff8c >2013-03-10 13:26:06,130 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [6d8106f2] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 13:26:06,130 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [298e1ca7] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 13:26:06,139 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-80) [28d4d994] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:26:06,141 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [28d4d994] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 17aa8000 >2013-03-10 13:26:06,142 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [28d4d994] FINISH, SetVdsStatusVDSCommand, log id: 17aa8000 >2013-03-10 13:26:06,148 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-80) [28d4d994] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 13:26:06,168 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-80) [7b41d828] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:26:06,170 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-80) [26059eec] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:26:06,172 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-80) [26059eec] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 13:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [68a361a0] Autorecovering 1 hosts >2013-03-10 13:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [68a361a0] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [73db6a29] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [73db6a29] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [73db6a29] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6297e493 >2013-03-10 13:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [73db6a29] FINISH, SetVdsStatusVDSCommand, log id: 6297e493 >2013-03-10 13:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-33) [73db6a29] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 312c8d42 >2013-03-10 13:30:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [73db6a29] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5344f0b0 >2013-03-10 13:30:00,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [73db6a29] FINISH, GetHardwareInfoVDSCommand, log id: 5344f0b0 >2013-03-10 13:30:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-33) [75c4e9f5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:30:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-33) [2e7368a4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:30:00,203 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-33) [2e7368a4] FINISH, ActivateVdsVDSCommand, log id: 312c8d42 >2013-03-10 13:30:00,222 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [2e7368a4] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:30:00,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-33) [2e7368a4] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 13:30:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [2e7368a4] Autorecovering 1 storage domains >2013-03-10 13:30:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [2e7368a4] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 13:30:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [2e915e53] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:30:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [2e915e53] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 1:30 PM >2013-03-10 13:30:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [2e915e53] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 1:30 PM >2013-03-10 13:30:00,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [7228cbd3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 51b9f78a >2013-03-10 13:30:00,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [7228cbd3] FINISH, GetHardwareInfoVDSCommand, log id: 51b9f78a >2013-03-10 13:30:00,420 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-1) [6d3073e5] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 13:30:00,424 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-1) [b7e2ed3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 13:30:00,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: c5a9935 >2013-03-10 13:30:33,837 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-74) [75083f49] Start refreshing all users data >2013-03-10 13:30:34,981 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-21) [722b2686] Attempting to update VMs/Templates Ovf. >2013-03-10 13:30:34,981 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-21) [722b2686] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 13:30:34,982 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-21) [722b2686] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 13:30:34,982 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-21) [722b2686] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 13:30:34,983 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-21) [722b2686] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 13:30:34,983 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-21) [722b2686] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 13:30:34,983 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-21) [722b2686] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 13:31:05,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 13:31:05,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:31:05,417 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 13:31:05,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: c5a9935 >2013-03-10 13:31:05,421 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [57762694] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:31:05,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [57762694] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 64b430c0 >2013-03-10 13:31:05,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [57762694] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 13:31:05,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [57762694] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 64b430c0 >2013-03-10 13:31:05,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [57762694] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 13:31:05,437 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [b7e2ed3] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 13:31:05,447 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-1) [238a8ad8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:31:05,449 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [238a8ad8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2a22e64e >2013-03-10 13:31:05,450 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [238a8ad8] FINISH, SetVdsStatusVDSCommand, log id: 2a22e64e >2013-03-10 13:31:05,456 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-1) [238a8ad8] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 13:31:05,477 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-1) [2d8cfd9e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:31:05,480 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-1) [763e1d0c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:31:05,481 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-1) [763e1d0c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 13:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [79bd4981] Autorecovering 1 hosts >2013-03-10 13:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [79bd4981] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [22ab2235] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [22ab2235] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [22ab2235] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7abc5dbd >2013-03-10 13:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [22ab2235] FINISH, SetVdsStatusVDSCommand, log id: 7abc5dbd >2013-03-10 13:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-20) [22ab2235] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4abdfa2b >2013-03-10 13:35:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [22ab2235] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 66ef875f >2013-03-10 13:35:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [22ab2235] FINISH, GetHardwareInfoVDSCommand, log id: 66ef875f >2013-03-10 13:35:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-20) [2a4ef34f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:35:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-20) [4bd79bfb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:35:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-20) [4bd79bfb] FINISH, ActivateVdsVDSCommand, log id: 4abdfa2b >2013-03-10 13:35:00,219 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [4bd79bfb] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:35:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-20) [4bd79bfb] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 13:35:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [4bd79bfb] Autorecovering 1 storage domains >2013-03-10 13:35:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [4bd79bfb] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 13:35:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [372fae9f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:35:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [372fae9f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 1:35 PM >2013-03-10 13:35:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [372fae9f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 1:35 PM >2013-03-10 13:35:02,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [fff7e03] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 8013e1f >2013-03-10 13:35:02,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [fff7e03] FINISH, GetHardwareInfoVDSCommand, log id: 8013e1f >2013-03-10 13:35:02,745 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-18) [5cc3dfa2] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 13:35:02,748 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-18) [41bf9233] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 13:35:02,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1c574d90 >2013-03-10 13:36:08,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 13:36:08,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:36:08,727 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 13:36:08,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 1c574d90 >2013-03-10 13:36:08,731 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [6331af26] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:36:08,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [6331af26] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 20d1c529 >2013-03-10 13:36:08,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [6331af26] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 13:36:08,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [6331af26] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 20d1c529 >2013-03-10 13:36:08,751 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [6331af26] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 13:36:08,751 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [41bf9233] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 13:36:08,761 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-18) [59c0e8b4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:36:08,762 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [59c0e8b4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 10c97e5a >2013-03-10 13:36:08,764 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [59c0e8b4] FINISH, SetVdsStatusVDSCommand, log id: 10c97e5a >2013-03-10 13:36:08,769 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [59c0e8b4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 13:36:08,791 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [721365e5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:36:08,794 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [1213812e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:36:08,795 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) [1213812e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 13:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [1ba1324f] Autorecovering 1 hosts >2013-03-10 13:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [1ba1324f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [711184cb] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [711184cb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [711184cb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 57b5fc7c >2013-03-10 13:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [711184cb] FINISH, SetVdsStatusVDSCommand, log id: 57b5fc7c >2013-03-10 13:40:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-87) [711184cb] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3e798be4 >2013-03-10 13:40:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [711184cb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7e19848b >2013-03-10 13:40:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [711184cb] FINISH, GetHardwareInfoVDSCommand, log id: 7e19848b >2013-03-10 13:40:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [6bcfa054] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:40:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [1a670a4e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:40:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-87) [1a670a4e] FINISH, ActivateVdsVDSCommand, log id: 3e798be4 >2013-03-10 13:40:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [1a670a4e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:40:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [1a670a4e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 13:40:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [1a670a4e] Autorecovering 1 storage domains >2013-03-10 13:40:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [1a670a4e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 13:40:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [45d010ba] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:40:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [45d010ba] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 1:40 PM >2013-03-10 13:40:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [45d010ba] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 1:40 PM >2013-03-10 13:40:02,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [7e0cc51a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 314d3e77 >2013-03-10 13:40:03,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [7e0cc51a] FINISH, GetHardwareInfoVDSCommand, log id: 314d3e77 >2013-03-10 13:40:03,037 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-99) [269b0c76] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 13:40:03,041 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-99) [6472866b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 13:40:03,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7903604c >2013-03-10 13:41:10,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 13:41:10,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:41:10,024 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 13:41:10,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 7903604c >2013-03-10 13:41:10,027 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [1f71a159] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:41:10,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [1f71a159] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 8055635 >2013-03-10 13:41:10,042 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [1f71a159] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 13:41:10,042 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [1f71a159] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 8055635 >2013-03-10 13:41:10,044 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [1f71a159] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 13:41:10,044 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [6472866b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 13:41:10,054 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-99) [4ab0f2d5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:41:10,055 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [4ab0f2d5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6d5cacdf >2013-03-10 13:41:10,056 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [4ab0f2d5] FINISH, SetVdsStatusVDSCommand, log id: 6d5cacdf >2013-03-10 13:41:10,063 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-99) [4ab0f2d5] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 13:41:10,084 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-99) [5dad9e4b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:41:10,087 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-99) [5cd7152c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:41:10,088 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-99) [5cd7152c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 13:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [1ccdcf1a] Autorecovering 1 hosts >2013-03-10 13:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [1ccdcf1a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [7f22a251] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [7f22a251] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [7f22a251] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7eeb876a >2013-03-10 13:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [7f22a251] FINISH, SetVdsStatusVDSCommand, log id: 7eeb876a >2013-03-10 13:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-64) [7f22a251] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3e87194e >2013-03-10 13:45:00,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [7f22a251] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 73da0548 >2013-03-10 13:45:00,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [7f22a251] FINISH, GetHardwareInfoVDSCommand, log id: 73da0548 >2013-03-10 13:45:00,778 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-64) [4adbe4d2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:45:00,780 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-64) [5af6b284] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:45:00,781 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-64) [5af6b284] FINISH, ActivateVdsVDSCommand, log id: 3e87194e >2013-03-10 13:45:00,801 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [5af6b284] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:45:00,802 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-64) [5af6b284] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 13:45:00,803 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [5af6b284] Autorecovering 1 storage domains >2013-03-10 13:45:00,803 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [5af6b284] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 13:45:00,804 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [4520582c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:45:00,804 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [4520582c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 1:45 PM >2013-03-10 13:45:00,806 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [4520582c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 1:45 PM >2013-03-10 13:45:01,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [1b38d5f2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 494ec3be >2013-03-10 13:45:01,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [1b38d5f2] FINISH, GetHardwareInfoVDSCommand, log id: 494ec3be >2013-03-10 13:45:01,327 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-83) [466b2e8d] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 13:45:01,331 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-83) [5c20fa5a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 13:45:01,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4666fe7a >2013-03-10 13:46:06,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 13:46:06,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:46:06,322 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 13:46:06,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 4666fe7a >2013-03-10 13:46:06,326 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [506a13e5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:46:06,339 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [506a13e5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 30583a9e >2013-03-10 13:46:06,340 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [506a13e5] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 13:46:06,340 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [506a13e5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 30583a9e >2013-03-10 13:46:06,342 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [506a13e5] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 13:46:06,342 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [5c20fa5a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 13:46:06,352 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-83) [4131e4af] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:46:06,354 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [4131e4af] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 29902246 >2013-03-10 13:46:06,355 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [4131e4af] FINISH, SetVdsStatusVDSCommand, log id: 29902246 >2013-03-10 13:46:06,361 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-83) [4131e4af] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 13:46:06,382 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-83) [28dbb7fd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:46:06,385 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-83) [2bb4e603] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:46:06,386 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-83) [2bb4e603] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 13:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [3719a4cc] Autorecovering 1 hosts >2013-03-10 13:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [3719a4cc] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-10) [7a7d631c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-10) [7a7d631c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [7a7d631c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: aaeb71 >2013-03-10 13:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [7a7d631c] FINISH, SetVdsStatusVDSCommand, log id: aaeb71 >2013-03-10 13:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-10) [7a7d631c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4d41c9b >2013-03-10 13:50:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [7a7d631c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: facdcd0 >2013-03-10 13:50:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [7a7d631c] FINISH, GetHardwareInfoVDSCommand, log id: facdcd0 >2013-03-10 13:50:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-10) [3ea900e3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:50:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-10) [7007f026] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:50:00,093 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-10) [7007f026] FINISH, ActivateVdsVDSCommand, log id: 4d41c9b >2013-03-10 13:50:00,112 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-10) [7007f026] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:50:00,113 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-10) [7007f026] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 13:50:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [7007f026] Autorecovering 1 storage domains >2013-03-10 13:50:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [7007f026] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 13:50:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-10) [1188e571] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:50:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-10) [1188e571] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 1:50 PM >2013-03-10 13:50:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-10) [1188e571] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 1:50 PM >2013-03-10 13:50:00,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [664159d3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 188f40ee >2013-03-10 13:50:00,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [664159d3] FINISH, GetHardwareInfoVDSCommand, log id: 188f40ee >2013-03-10 13:50:00,626 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-84) [4412579b] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 13:50:00,630 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-84) [52573057] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 13:50:00,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 42217b5d >2013-03-10 13:51:06,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 13:51:06,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:51:06,615 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 13:51:06,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 42217b5d >2013-03-10 13:51:06,619 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [c2cdae7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:51:06,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [c2cdae7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2e07d13c >2013-03-10 13:51:06,633 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [c2cdae7] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 13:51:06,633 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [c2cdae7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2e07d13c >2013-03-10 13:51:06,635 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [c2cdae7] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 13:51:06,635 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [52573057] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 13:51:06,645 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-84) [4b8ccf8f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:51:06,646 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [4b8ccf8f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 65ae583c >2013-03-10 13:51:06,647 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [4b8ccf8f] FINISH, SetVdsStatusVDSCommand, log id: 65ae583c >2013-03-10 13:51:06,653 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-84) [4b8ccf8f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 13:51:06,675 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-84) [85c7641] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:51:06,678 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-84) [6887909e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:51:06,679 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-84) [6887909e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 13:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [70a3eb0a] Autorecovering 1 hosts >2013-03-10 13:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [70a3eb0a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [7b121405] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [7b121405] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [7b121405] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 38418ea4 >2013-03-10 13:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [7b121405] FINISH, SetVdsStatusVDSCommand, log id: 38418ea4 >2013-03-10 13:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [7b121405] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 72f44d6f >2013-03-10 13:55:00,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [7b121405] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5fc5577a >2013-03-10 13:55:00,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [7b121405] FINISH, GetHardwareInfoVDSCommand, log id: 5fc5577a >2013-03-10 13:55:00,470 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [d6de43f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:55:00,472 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [2013e3be] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:55:00,473 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [2013e3be] FINISH, ActivateVdsVDSCommand, log id: 72f44d6f >2013-03-10 13:55:00,493 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [2013e3be] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 13:55:00,497 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [2013e3be] Autorecovering 1 storage domains >2013-03-10 13:55:00,497 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [2013e3be] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 13:55:00,497 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [8897c9b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:55:00,498 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [8897c9b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 1:55 PM >2013-03-10 13:55:00,500 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [8897c9b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 1:55 PM >2013-03-10 13:55:00,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [a9105b1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 38241670 >2013-03-10 13:55:00,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [a9105b1] FINISH, GetHardwareInfoVDSCommand, log id: 38241670 >2013-03-10 13:55:00,958 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-19) [4298af47] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 13:55:00,961 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-19) [2764863d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 13:55:00,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3fc66f84 >2013-03-10 13:56:07,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 13:56:07,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 13:56:07,029 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 13:56:07,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 3fc66f84 >2013-03-10 13:56:07,032 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [6e45c899] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 13:56:07,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6e45c899] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 36f90006 >2013-03-10 13:56:07,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [6e45c899] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 13:56:07,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6e45c899] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 36f90006 >2013-03-10 13:56:07,051 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [2764863d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 13:56:07,060 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-19) [41b663b2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:56:07,062 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [41b663b2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 114b78b0 >2013-03-10 13:56:07,063 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [41b663b2] FINISH, SetVdsStatusVDSCommand, log id: 114b78b0 >2013-03-10 13:56:07,069 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-19) [41b663b2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 13:56:07,089 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-19) [26cc8e8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:56:07,091 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-19) [266664c7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 13:56:07,093 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-19) [266664c7] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 14:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [23a1f3ef] Autorecovering 1 hosts >2013-03-10 14:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [23a1f3ef] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [21682da6] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [21682da6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [21682da6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5c0ad43 >2013-03-10 14:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [21682da6] FINISH, SetVdsStatusVDSCommand, log id: 5c0ad43 >2013-03-10 14:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [21682da6] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 485c8657 >2013-03-10 14:00:00,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [21682da6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5a3fac22 >2013-03-10 14:00:00,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [21682da6] FINISH, GetHardwareInfoVDSCommand, log id: 5a3fac22 >2013-03-10 14:00:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-16) [b232bb2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:00:00,208 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-16) [3e94a060] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:00:00,209 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [3e94a060] FINISH, ActivateVdsVDSCommand, log id: 485c8657 >2013-03-10 14:00:00,229 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [3e94a060] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:00:00,229 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-16) [3e94a060] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 14:00:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [3e94a060] Autorecovering 1 storage domains >2013-03-10 14:00:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [3e94a060] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 14:00:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [68ecd198] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:00:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [68ecd198] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 2:00 PM >2013-03-10 14:00:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [68ecd198] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 2:00 PM >2013-03-10 14:00:01,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [6af286d9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: a96f0d3 >2013-03-10 14:00:01,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [6af286d9] FINISH, GetHardwareInfoVDSCommand, log id: a96f0d3 >2013-03-10 14:00:01,336 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-88) [7076bc9b] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 14:00:01,339 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-88) [45743506] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 14:00:01,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3070117 >2013-03-10 14:01:08,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 14:01:08,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:01:08,365 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 14:01:08,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 3070117 >2013-03-10 14:01:08,371 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [39ee062b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:01:08,385 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [39ee062b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6e3763b7 >2013-03-10 14:01:08,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [39ee062b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 14:01:08,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [39ee062b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6e3763b7 >2013-03-10 14:01:08,388 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [39ee062b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 14:01:08,388 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [45743506] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 14:01:08,398 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-88) [70b5ab41] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:01:08,400 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [70b5ab41] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6193d13c >2013-03-10 14:01:08,401 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [70b5ab41] FINISH, SetVdsStatusVDSCommand, log id: 6193d13c >2013-03-10 14:01:08,406 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-88) [70b5ab41] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 14:01:08,427 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-88) [52606e61] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:01:08,430 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-88) [2c4ffb2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:01:08,431 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-88) [2c4ffb2] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 14:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [1902f6b0] Autorecovering 1 hosts >2013-03-10 14:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [1902f6b0] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [311a86c8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [311a86c8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [311a86c8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 81be3d5 >2013-03-10 14:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [311a86c8] FINISH, SetVdsStatusVDSCommand, log id: 81be3d5 >2013-03-10 14:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-68) [311a86c8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 728b8c7f >2013-03-10 14:05:00,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [311a86c8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 56a04666 >2013-03-10 14:05:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [311a86c8] FINISH, GetHardwareInfoVDSCommand, log id: 56a04666 >2013-03-10 14:05:00,183 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-68) [6e70ac72] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:05:00,186 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-68) [7e165c07] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:05:00,187 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-68) [7e165c07] FINISH, ActivateVdsVDSCommand, log id: 728b8c7f >2013-03-10 14:05:00,206 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [7e165c07] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:05:00,207 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-68) [7e165c07] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 14:05:00,208 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [7e165c07] Autorecovering 1 storage domains >2013-03-10 14:05:00,208 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [7e165c07] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 14:05:00,209 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [3bff3564] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:05:00,209 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [3bff3564] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 2:05 PM >2013-03-10 14:05:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [3bff3564] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 2:05 PM >2013-03-10 14:05:02,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [43d6587e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 16ab99c8 >2013-03-10 14:05:02,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [43d6587e] FINISH, GetHardwareInfoVDSCommand, log id: 16ab99c8 >2013-03-10 14:05:02,672 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-2) [6193d99d] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 14:05:02,676 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-2) [59abe681] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 14:05:02,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6a43a49 >2013-03-10 14:06:07,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 14:06:07,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:06:07,673 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 14:06:07,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 6a43a49 >2013-03-10 14:06:07,677 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [c48ca22] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:06:07,691 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [c48ca22] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3e5f163 >2013-03-10 14:06:07,691 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [c48ca22] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 14:06:07,691 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [c48ca22] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3e5f163 >2013-03-10 14:06:07,693 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [c48ca22] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 14:06:07,693 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [59abe681] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 14:06:07,703 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-2) [32f9d3c2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:06:07,704 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [32f9d3c2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 57cbc609 >2013-03-10 14:06:07,705 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [32f9d3c2] FINISH, SetVdsStatusVDSCommand, log id: 57cbc609 >2013-03-10 14:06:07,711 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [32f9d3c2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 14:06:07,733 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [55b40e84] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:06:07,736 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [4d280997] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:06:07,737 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-2) [4d280997] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 14:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [248f5a32] Autorecovering 1 hosts >2013-03-10 14:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [248f5a32] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [10df0d08] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [10df0d08] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [10df0d08] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 38cc9813 >2013-03-10 14:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [10df0d08] FINISH, SetVdsStatusVDSCommand, log id: 38cc9813 >2013-03-10 14:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-32) [10df0d08] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 13b76a12 >2013-03-10 14:10:00,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [10df0d08] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 69d97e5c >2013-03-10 14:10:00,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [10df0d08] FINISH, GetHardwareInfoVDSCommand, log id: 69d97e5c >2013-03-10 14:10:00,276 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-32) [2bcd38d7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:10:00,278 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-32) [1d7f99f9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:10:00,280 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-32) [1d7f99f9] FINISH, ActivateVdsVDSCommand, log id: 13b76a12 >2013-03-10 14:10:00,299 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [1d7f99f9] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:10:00,300 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-32) [1d7f99f9] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 14:10:00,301 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [1d7f99f9] Autorecovering 1 storage domains >2013-03-10 14:10:00,301 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [1d7f99f9] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 14:10:00,302 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [4cc072b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:10:00,302 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [4cc072b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 2:10 PM >2013-03-10 14:10:00,305 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [4cc072b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 2:10 PM >2013-03-10 14:10:01,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [331e8d6f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 470da471 >2013-03-10 14:10:01,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [331e8d6f] FINISH, GetHardwareInfoVDSCommand, log id: 470da471 >2013-03-10 14:10:01,976 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-22) [3b9b377c] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 14:10:01,979 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-22) [6099fa0f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 14:10:01,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: f8b635b >2013-03-10 14:11:08,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 14:11:08,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:11:08,989 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 14:11:08,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: f8b635b >2013-03-10 14:11:08,993 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [513d2d30] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:11:09,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [513d2d30] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 53ee4959 >2013-03-10 14:11:09,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [513d2d30] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 14:11:09,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [513d2d30] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 53ee4959 >2013-03-10 14:11:09,016 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [513d2d30] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 14:11:09,017 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [6099fa0f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 14:11:09,026 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-22) [2c92c0f4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:11:09,027 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [2c92c0f4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 72013ecf >2013-03-10 14:11:09,029 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [2c92c0f4] FINISH, SetVdsStatusVDSCommand, log id: 72013ecf >2013-03-10 14:11:09,034 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [2c92c0f4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 14:11:09,055 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [13697c5d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:11:09,057 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [3bacf37] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:11:09,059 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-22) [3bacf37] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 14:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [1ce1a6db] Autorecovering 1 hosts >2013-03-10 14:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [1ce1a6db] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-91) [7c7bdc8c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-91) [7c7bdc8c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [7c7bdc8c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3e3b7a14 >2013-03-10 14:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [7c7bdc8c] FINISH, SetVdsStatusVDSCommand, log id: 3e3b7a14 >2013-03-10 14:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-91) [7c7bdc8c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 28b40f94 >2013-03-10 14:15:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [7c7bdc8c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 78a71b78 >2013-03-10 14:15:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [7c7bdc8c] FINISH, GetHardwareInfoVDSCommand, log id: 78a71b78 >2013-03-10 14:15:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-91) [388bec2b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:15:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-91) [3beea30f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:15:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-91) [3beea30f] FINISH, ActivateVdsVDSCommand, log id: 28b40f94 >2013-03-10 14:15:00,225 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-91) [3beea30f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:15:00,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-91) [3beea30f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 14:15:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [3beea30f] Autorecovering 1 storage domains >2013-03-10 14:15:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [3beea30f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 14:15:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-91) [1510a5b2] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:15:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-91) [1510a5b2] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 2:15 PM >2013-03-10 14:15:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-91) [1510a5b2] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 2:15 PM >2013-03-10 14:15:00,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [34cfa5f2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2505075a >2013-03-10 14:15:00,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [34cfa5f2] FINISH, GetHardwareInfoVDSCommand, log id: 2505075a >2013-03-10 14:15:00,309 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-89) [579f1923] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 14:15:00,312 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-89) [f9e92a9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 14:15:00,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1b2868d7 >2013-03-10 14:16:05,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 14:16:05,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:16:05,283 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 14:16:05,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 1b2868d7 >2013-03-10 14:16:05,287 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [5f86717] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:16:05,301 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [5f86717] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6519ff33 >2013-03-10 14:16:05,301 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [5f86717] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 14:16:05,301 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [5f86717] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6519ff33 >2013-03-10 14:16:05,303 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [5f86717] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 14:16:05,303 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [f9e92a9] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 14:16:05,313 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-89) [a6e6f06] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:16:05,314 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [a6e6f06] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2c097f2 >2013-03-10 14:16:05,316 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [a6e6f06] FINISH, SetVdsStatusVDSCommand, log id: 2c097f2 >2013-03-10 14:16:05,321 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-89) [a6e6f06] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 14:16:05,342 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-89) [16ebdf13] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:16:05,345 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-89) [2fb329b2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:16:05,346 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-89) [2fb329b2] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 14:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [770cf360] Autorecovering 1 hosts >2013-03-10 14:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [770cf360] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [282225f2] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [282225f2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [282225f2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 219dde2e >2013-03-10 14:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [282225f2] FINISH, SetVdsStatusVDSCommand, log id: 219dde2e >2013-03-10 14:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [282225f2] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 44290868 >2013-03-10 14:20:00,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [282225f2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 767226eb >2013-03-10 14:20:00,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [282225f2] FINISH, GetHardwareInfoVDSCommand, log id: 767226eb >2013-03-10 14:20:00,408 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-79) [55defbf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:20:00,419 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-79) [275a4d3a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:20:00,420 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [275a4d3a] FINISH, ActivateVdsVDSCommand, log id: 44290868 >2013-03-10 14:20:00,440 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [275a4d3a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:20:00,441 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-79) [275a4d3a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 14:20:00,442 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [275a4d3a] Autorecovering 1 storage domains >2013-03-10 14:20:00,442 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [275a4d3a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 14:20:00,442 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [1e536f05] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:20:00,443 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [1e536f05] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 2:20 PM >2013-03-10 14:20:00,445 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [1e536f05] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 2:20 PM >2013-03-10 14:20:02,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [1c455426] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7060d785 >2013-03-10 14:20:02,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [1c455426] FINISH, GetHardwareInfoVDSCommand, log id: 7060d785 >2013-03-10 14:20:02,589 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-26) [bd18a31] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 14:20:02,592 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-26) [71ac4bd3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 14:20:02,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1f5df327 >2013-03-10 14:21:07,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 14:21:07,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:21:07,602 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 14:21:07,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 1f5df327 >2013-03-10 14:21:07,606 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [68fab559] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:21:07,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [68fab559] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 42504082 >2013-03-10 14:21:07,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [68fab559] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 14:21:07,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [68fab559] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 42504082 >2013-03-10 14:21:07,622 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [68fab559] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 14:21:07,622 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [71ac4bd3] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 14:21:07,632 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-26) [735824fc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:21:07,633 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [735824fc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1d176741 >2013-03-10 14:21:07,635 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [735824fc] FINISH, SetVdsStatusVDSCommand, log id: 1d176741 >2013-03-10 14:21:07,640 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [735824fc] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 14:21:07,662 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [4bb277b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:21:07,665 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [5663ec8f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:21:07,666 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-26) [5663ec8f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 14:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [2e6cd64] Autorecovering 1 hosts >2013-03-10 14:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [2e6cd64] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-39) [8b50bb8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-39) [8b50bb8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [8b50bb8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6b4d068c >2013-03-10 14:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [8b50bb8] FINISH, SetVdsStatusVDSCommand, log id: 6b4d068c >2013-03-10 14:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-39) [8b50bb8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 69b73df4 >2013-03-10 14:25:00,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [8b50bb8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3d0e5ceb >2013-03-10 14:25:00,559 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [8b50bb8] FINISH, GetHardwareInfoVDSCommand, log id: 3d0e5ceb >2013-03-10 14:25:00,575 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-39) [1528c7c5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:25:00,578 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-39) [3430ffa7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:25:00,579 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-39) [3430ffa7] FINISH, ActivateVdsVDSCommand, log id: 69b73df4 >2013-03-10 14:25:00,598 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-39) [3430ffa7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:25:00,599 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-39) [3430ffa7] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 14:25:00,600 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [3430ffa7] Autorecovering 1 storage domains >2013-03-10 14:25:00,600 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [3430ffa7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 14:25:00,601 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-39) [497893c7] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:25:00,601 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-39) [497893c7] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 2:25 PM >2013-03-10 14:25:00,604 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-39) [497893c7] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 2:25 PM >2013-03-10 14:25:01,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [40d50ad7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 19d7f290 >2013-03-10 14:25:01,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [40d50ad7] FINISH, GetHardwareInfoVDSCommand, log id: 19d7f290 >2013-03-10 14:25:01,907 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-66) [73f7aeca] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 14:25:01,911 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-66) [46ab9406] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 14:25:01,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 726853f7 >2013-03-10 14:26:09,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 14:26:09,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:26:09,956 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 14:26:09,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 726853f7 >2013-03-10 14:26:09,959 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [6f6cb896] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:26:09,973 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [6f6cb896] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 12402333 >2013-03-10 14:26:09,973 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [6f6cb896] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 14:26:09,973 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [6f6cb896] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 12402333 >2013-03-10 14:26:09,975 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [6f6cb896] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 14:26:09,975 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [46ab9406] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 14:26:09,985 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-66) [69db8d2b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:26:09,987 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [69db8d2b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 104f98ba >2013-03-10 14:26:09,988 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [69db8d2b] FINISH, SetVdsStatusVDSCommand, log id: 104f98ba >2013-03-10 14:26:09,993 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-66) [69db8d2b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 14:26:10,014 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-66) [5ca86164] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:26:10,016 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-66) [39c82f66] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:26:10,017 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-66) [39c82f66] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 14:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [7161258c] Autorecovering 1 hosts >2013-03-10 14:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [7161258c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [5afe7a1a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [5afe7a1a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [5afe7a1a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 9601406 >2013-03-10 14:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [5afe7a1a] FINISH, SetVdsStatusVDSCommand, log id: 9601406 >2013-03-10 14:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-57) [5afe7a1a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5db82a11 >2013-03-10 14:30:00,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [5afe7a1a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4f10d00d >2013-03-10 14:30:00,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [5afe7a1a] FINISH, GetHardwareInfoVDSCommand, log id: 4f10d00d >2013-03-10 14:30:00,098 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-57) [7acb4d10] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:30:00,100 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-57) [62cc1867] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:30:00,101 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-57) [62cc1867] FINISH, ActivateVdsVDSCommand, log id: 5db82a11 >2013-03-10 14:30:00,121 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [62cc1867] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:30:00,121 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-57) [62cc1867] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 14:30:00,123 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [62cc1867] Autorecovering 1 storage domains >2013-03-10 14:30:00,123 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [62cc1867] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 14:30:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [77649e34] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:30:00,124 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [77649e34] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 2:30 PM >2013-03-10 14:30:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [77649e34] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 2:30 PM >2013-03-10 14:30:01,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [d1e2a77] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 21376d8d >2013-03-10 14:30:01,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [d1e2a77] FINISH, GetHardwareInfoVDSCommand, log id: 21376d8d >2013-03-10 14:30:01,259 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-56) [48e5c24] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 14:30:01,263 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-56) [305852f8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 14:30:01,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: f2ac642 >2013-03-10 14:30:33,839 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-78) [5081fd3d] Start refreshing all users data >2013-03-10 14:30:34,983 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Attempting to update VMs/Templates Ovf. >2013-03-10 14:30:34,984 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 14:30:34,985 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 14:30:34,985 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 14:30:34,986 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 14:30:34,986 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 14:30:34,987 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [552c831b] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 14:31:08,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 14:31:08,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:31:08,302 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 14:31:08,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: f2ac642 >2013-03-10 14:31:08,306 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [3ef74733] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:31:08,319 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [3ef74733] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1272db2f >2013-03-10 14:31:08,319 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [3ef74733] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 14:31:08,319 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [3ef74733] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1272db2f >2013-03-10 14:31:08,321 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [3ef74733] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 14:31:08,322 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [305852f8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 14:31:08,331 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-56) [265a95ae] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:31:08,333 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [265a95ae] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6d703893 >2013-03-10 14:31:08,334 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [265a95ae] FINISH, SetVdsStatusVDSCommand, log id: 6d703893 >2013-03-10 14:31:08,340 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-56) [265a95ae] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 14:31:08,389 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-56) [60eeadac] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:31:08,391 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-56) [739675ba] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:31:08,393 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-56) [739675ba] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 14:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [20753663] Autorecovering 1 hosts >2013-03-10 14:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [20753663] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [c9cae05] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [c9cae05] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [c9cae05] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 75b21767 >2013-03-10 14:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [c9cae05] FINISH, SetVdsStatusVDSCommand, log id: 75b21767 >2013-03-10 14:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-35) [c9cae05] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 583e88de >2013-03-10 14:35:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [c9cae05] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 14eef0ce >2013-03-10 14:35:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [c9cae05] FINISH, GetHardwareInfoVDSCommand, log id: 14eef0ce >2013-03-10 14:35:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-35) [642e3778] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:35:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-35) [3ee4f899] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:35:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-35) [3ee4f899] FINISH, ActivateVdsVDSCommand, log id: 583e88de >2013-03-10 14:35:00,111 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [3ee4f899] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:35:00,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-35) [3ee4f899] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 14:35:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [3ee4f899] Autorecovering 1 storage domains >2013-03-10 14:35:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [3ee4f899] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 14:35:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [3b5a8bb6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:35:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [3b5a8bb6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 2:35 PM >2013-03-10 14:35:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [3b5a8bb6] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 2:35 PM >2013-03-10 14:35:02,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [368aa39f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: b58cfa9 >2013-03-10 14:35:02,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [368aa39f] FINISH, GetHardwareInfoVDSCommand, log id: b58cfa9 >2013-03-10 14:35:02,636 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-28) [285e7668] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 14:35:02,640 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-28) [48f8eb57] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 14:35:02,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: f3ac4aa >2013-03-10 14:36:10,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 14:36:10,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:36:10,654 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 14:36:10,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: f3ac4aa >2013-03-10 14:36:10,658 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [33eead1a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:36:10,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [33eead1a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6c4dc127 >2013-03-10 14:36:10,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [33eead1a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 14:36:10,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [33eead1a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6c4dc127 >2013-03-10 14:36:10,675 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [33eead1a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 14:36:10,676 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [48f8eb57] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 14:36:10,686 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-28) [58a94938] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:36:10,687 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [58a94938] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1862a760 >2013-03-10 14:36:10,688 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [58a94938] FINISH, SetVdsStatusVDSCommand, log id: 1862a760 >2013-03-10 14:36:10,694 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-28) [58a94938] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 14:36:10,725 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-28) [24e8cdfb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:36:10,728 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-28) [775e1638] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:36:10,729 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-28) [775e1638] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 14:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [e31a467] Autorecovering 1 hosts >2013-03-10 14:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [e31a467] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [6938b787] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [6938b787] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [6938b787] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 709f32b2 >2013-03-10 14:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [6938b787] FINISH, SetVdsStatusVDSCommand, log id: 709f32b2 >2013-03-10 14:40:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-13) [6938b787] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5a3d3181 >2013-03-10 14:40:00,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [6938b787] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3742397d >2013-03-10 14:40:00,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [6938b787] FINISH, GetHardwareInfoVDSCommand, log id: 3742397d >2013-03-10 14:40:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-13) [38844eb0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:40:00,096 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-13) [4a79804e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:40:00,097 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-13) [4a79804e] FINISH, ActivateVdsVDSCommand, log id: 5a3d3181 >2013-03-10 14:40:00,117 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [4a79804e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:40:00,118 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-13) [4a79804e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 14:40:00,119 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [4a79804e] Autorecovering 1 storage domains >2013-03-10 14:40:00,119 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [4a79804e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 14:40:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [449a10e6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:40:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [449a10e6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 2:40 PM >2013-03-10 14:40:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [449a10e6] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 2:40 PM >2013-03-10 14:40:01,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [296259f6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 56108dd3 >2013-03-10 14:40:01,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [296259f6] FINISH, GetHardwareInfoVDSCommand, log id: 56108dd3 >2013-03-10 14:40:01,970 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-17) [4b2dda0d] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 14:40:01,973 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-17) [40ac3deb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 14:40:01,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5c82e7ed >2013-03-10 14:41:09,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 14:41:09,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:41:09,007 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 14:41:09,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 5c82e7ed >2013-03-10 14:41:09,010 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [5dcfa42] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:41:09,024 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [5dcfa42] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 569e6a6 >2013-03-10 14:41:09,024 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [5dcfa42] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 14:41:09,024 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [5dcfa42] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 569e6a6 >2013-03-10 14:41:09,026 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [5dcfa42] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 14:41:09,027 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [40ac3deb] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 14:41:09,036 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-17) [68b8beb6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:41:09,037 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [68b8beb6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 8aa39db >2013-03-10 14:41:09,038 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [68b8beb6] FINISH, SetVdsStatusVDSCommand, log id: 8aa39db >2013-03-10 14:41:09,044 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [68b8beb6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 14:41:09,065 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [788f6391] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:41:09,068 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [2c221c0b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:41:09,069 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-17) [2c221c0b] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 14:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) [2ab50aab] Autorecovering 1 hosts >2013-03-10 14:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) [2ab50aab] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-11) [3f7864f3] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-11) [3f7864f3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [3f7864f3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3e6981b2 >2013-03-10 14:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [3f7864f3] FINISH, SetVdsStatusVDSCommand, log id: 3e6981b2 >2013-03-10 14:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-11) [3f7864f3] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: cb9802f >2013-03-10 14:45:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [3f7864f3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2d2e91bf >2013-03-10 14:45:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [3f7864f3] FINISH, GetHardwareInfoVDSCommand, log id: 2d2e91bf >2013-03-10 14:45:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-11) [9c3aa90] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:45:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-11) [96550ce] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:45:00,095 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-11) [96550ce] FINISH, ActivateVdsVDSCommand, log id: cb9802f >2013-03-10 14:45:00,114 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-11) [96550ce] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:45:00,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-11) [96550ce] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 14:45:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) [96550ce] Autorecovering 1 storage domains >2013-03-10 14:45:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) [96550ce] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 14:45:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-11) [3a3123f8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:45:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-11) [3a3123f8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 2:45 PM >2013-03-10 14:45:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-11) [3a3123f8] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 2:45 PM >2013-03-10 14:45:00,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [2e915e53] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 423c0ae2 >2013-03-10 14:45:00,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [2e915e53] FINISH, GetHardwareInfoVDSCommand, log id: 423c0ae2 >2013-03-10 14:45:00,312 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-33) [184af72b] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 14:45:00,316 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-33) [2216f99] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 14:45:00,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 487fd1b7 >2013-03-10 14:46:05,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 14:46:05,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:46:05,311 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 14:46:05,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 487fd1b7 >2013-03-10 14:46:05,314 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [7fd737c5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:46:05,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [7fd737c5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7b52974d >2013-03-10 14:46:05,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [7fd737c5] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 14:46:05,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [7fd737c5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7b52974d >2013-03-10 14:46:05,330 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [7fd737c5] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 14:46:05,330 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [2216f99] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 14:46:05,340 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-33) [3657416] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:46:05,341 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [3657416] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1ca4475f >2013-03-10 14:46:05,342 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [3657416] FINISH, SetVdsStatusVDSCommand, log id: 1ca4475f >2013-03-10 14:46:05,380 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-33) [3657416] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 14:46:05,401 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-33) [5b429e59] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:46:05,403 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-33) [681b38ac] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:46:05,405 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-33) [681b38ac] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 14:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [2aa7b284] Autorecovering 1 hosts >2013-03-10 14:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [2aa7b284] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-95) [21a7848d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-95) [21a7848d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [21a7848d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1edd1433 >2013-03-10 14:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [21a7848d] FINISH, SetVdsStatusVDSCommand, log id: 1edd1433 >2013-03-10 14:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-95) [21a7848d] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 24d7c31e >2013-03-10 14:50:00,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [21a7848d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4e977913 >2013-03-10 14:50:00,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [21a7848d] FINISH, GetHardwareInfoVDSCommand, log id: 4e977913 >2013-03-10 14:50:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-95) [7c94edca] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:50:00,095 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-95) [172409d0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:50:00,096 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-95) [172409d0] FINISH, ActivateVdsVDSCommand, log id: 24d7c31e >2013-03-10 14:50:00,115 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-95) [172409d0] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:50:00,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-95) [172409d0] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 14:50:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [172409d0] Autorecovering 1 storage domains >2013-03-10 14:50:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [172409d0] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 14:50:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-95) [4c5ace3e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:50:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-95) [4c5ace3e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 2:50 PM >2013-03-10 14:50:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-95) [4c5ace3e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 2:50 PM >2013-03-10 14:50:02,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [77e00969] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7979de91 >2013-03-10 14:50:02,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [77e00969] FINISH, GetHardwareInfoVDSCommand, log id: 7979de91 >2013-03-10 14:50:02,652 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-48) [5d67b86b] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 14:50:02,655 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-48) [45fce476] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 14:50:02,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 19a529ea >2013-03-10 14:51:07,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 14:51:07,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:51:07,652 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 14:51:07,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 19a529ea >2013-03-10 14:51:07,655 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [7fbb3bed] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:51:07,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [7fbb3bed] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 29fb7a58 >2013-03-10 14:51:07,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [7fbb3bed] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 14:51:07,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [7fbb3bed] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 29fb7a58 >2013-03-10 14:51:07,672 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [7fbb3bed] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 14:51:07,672 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [45fce476] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 14:51:07,681 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-48) [42bcc755] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:51:07,683 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [42bcc755] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4a2646b0 >2013-03-10 14:51:07,684 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [42bcc755] FINISH, SetVdsStatusVDSCommand, log id: 4a2646b0 >2013-03-10 14:51:07,690 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-48) [42bcc755] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 14:51:07,711 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-48) [f7184db] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:51:07,714 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-48) [689413c1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:51:07,715 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-48) [689413c1] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 14:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [23ffeabc] Autorecovering 1 hosts >2013-03-10 14:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [23ffeabc] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [70e182dd] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [70e182dd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [70e182dd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 15788bb7 >2013-03-10 14:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [70e182dd] FINISH, SetVdsStatusVDSCommand, log id: 15788bb7 >2013-03-10 14:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [70e182dd] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2c5d9638 >2013-03-10 14:55:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [70e182dd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 73aa765f >2013-03-10 14:55:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [70e182dd] FINISH, GetHardwareInfoVDSCommand, log id: 73aa765f >2013-03-10 14:55:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-96) [25301ef0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:55:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-96) [50e0f3e2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:55:00,091 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [50e0f3e2] FINISH, ActivateVdsVDSCommand, log id: 2c5d9638 >2013-03-10 14:55:00,115 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [50e0f3e2] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 14:55:00,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-96) [50e0f3e2] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 14:55:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [50e0f3e2] Autorecovering 1 storage domains >2013-03-10 14:55:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [50e0f3e2] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 14:55:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [2a8077ba] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:55:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [2a8077ba] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 2:55 PM >2013-03-10 14:55:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [2a8077ba] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 2:55 PM >2013-03-10 14:55:01,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [47e90346] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7e25f5f5 >2013-03-10 14:55:01,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [47e90346] FINISH, GetHardwareInfoVDSCommand, log id: 7e25f5f5 >2013-03-10 14:55:01,963 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-4) [3de0cb0d] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 14:55:01,966 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-4) [6146ec7b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 14:55:01,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 47182528 >2013-03-10 14:56:10,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 14:56:10,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 14:56:10,019 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 14:56:10,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 47182528 >2013-03-10 14:56:10,023 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [75a43728] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 14:56:10,039 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [75a43728] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 653aa454 >2013-03-10 14:56:10,039 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [75a43728] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 14:56:10,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [75a43728] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 653aa454 >2013-03-10 14:56:10,044 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [6146ec7b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 14:56:10,054 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-4) [3d7610e1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:56:10,055 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [3d7610e1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6172566a >2013-03-10 14:56:10,056 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [3d7610e1] FINISH, SetVdsStatusVDSCommand, log id: 6172566a >2013-03-10 14:56:10,062 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-4) [3d7610e1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 14:56:10,083 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-4) [3cd30650] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:56:10,086 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-4) [5329e9ba] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 14:56:10,087 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-4) [5329e9ba] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 15:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [5c93cf3e] Autorecovering 1 hosts >2013-03-10 15:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [5c93cf3e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [50af24b8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [50af24b8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [50af24b8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 76566c52 >2013-03-10 15:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [50af24b8] FINISH, SetVdsStatusVDSCommand, log id: 76566c52 >2013-03-10 15:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-70) [50af24b8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 55969ef5 >2013-03-10 15:00:00,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [50af24b8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 783531c >2013-03-10 15:00:00,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [50af24b8] FINISH, GetHardwareInfoVDSCommand, log id: 783531c >2013-03-10 15:00:00,364 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-70) [2a9db91e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:00:00,366 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-70) [621cab39] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:00:00,367 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-70) [621cab39] FINISH, ActivateVdsVDSCommand, log id: 55969ef5 >2013-03-10 15:00:00,387 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [621cab39] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:00:00,388 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-70) [621cab39] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 15:00:00,389 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [621cab39] Autorecovering 1 storage domains >2013-03-10 15:00:00,389 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [621cab39] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 15:00:00,390 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [2bda9e33] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:00:00,390 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [2bda9e33] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 3:00 PM >2013-03-10 15:00:00,392 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [2bda9e33] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 3:00 PM >2013-03-10 15:00:01,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [3bacf37] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 780481f1 >2013-03-10 15:00:01,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [3bacf37] FINISH, GetHardwareInfoVDSCommand, log id: 780481f1 >2013-03-10 15:00:01,329 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-22) [71b942dc] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 15:00:01,332 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-22) [26c12f36] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 15:00:01,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7cb2ad9d >2013-03-10 15:01:08,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 15:01:08,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:01:08,360 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 15:01:08,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 7cb2ad9d >2013-03-10 15:01:08,364 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [2b8c57ca] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:01:08,378 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [2b8c57ca] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 16e99ed >2013-03-10 15:01:08,378 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [2b8c57ca] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 15:01:08,378 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [2b8c57ca] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 16e99ed >2013-03-10 15:01:08,380 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [2b8c57ca] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 15:01:08,380 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [26c12f36] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 15:01:08,390 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-22) [7349cc0d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:01:08,391 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [7349cc0d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 353f1ac1 >2013-03-10 15:01:08,392 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [7349cc0d] FINISH, SetVdsStatusVDSCommand, log id: 353f1ac1 >2013-03-10 15:01:08,398 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [7349cc0d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 15:01:08,419 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [1baada6d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:01:08,421 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [6040c924] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:01:08,423 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-22) [6040c924] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 15:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [1188e571] Autorecovering 1 hosts >2013-03-10 15:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [1188e571] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-10) [550a2846] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-10) [550a2846] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [550a2846] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3c953db9 >2013-03-10 15:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [550a2846] FINISH, SetVdsStatusVDSCommand, log id: 3c953db9 >2013-03-10 15:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-10) [550a2846] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 55e31ccc >2013-03-10 15:05:00,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [550a2846] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5aaaa5f6 >2013-03-10 15:05:00,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [550a2846] FINISH, GetHardwareInfoVDSCommand, log id: 5aaaa5f6 >2013-03-10 15:05:00,212 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-10) [1b60cafe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:05:00,214 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-10) [795b6d8a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:05:00,216 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-10) [795b6d8a] FINISH, ActivateVdsVDSCommand, log id: 55e31ccc >2013-03-10 15:05:00,235 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-10) [795b6d8a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:05:00,235 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-10) [795b6d8a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 15:05:00,237 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [795b6d8a] Autorecovering 1 storage domains >2013-03-10 15:05:00,237 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [795b6d8a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 15:05:00,237 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-10) [6566b16c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:05:00,238 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-10) [6566b16c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 3:05 PM >2013-03-10 15:05:00,240 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-10) [6566b16c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 3:05 PM >2013-03-10 15:05:02,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [5081fd3d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 258deb34 >2013-03-10 15:05:02,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [5081fd3d] FINISH, GetHardwareInfoVDSCommand, log id: 258deb34 >2013-03-10 15:05:02,666 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-78) [384c96f] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 15:05:02,670 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-78) [1e9afe3e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 15:05:02,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 111c1c3f >2013-03-10 15:06:09,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 15:06:09,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:06:09,683 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 15:06:09,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 111c1c3f >2013-03-10 15:06:09,687 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [61f94267] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:06:09,701 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [61f94267] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 25f373bf >2013-03-10 15:06:09,701 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [61f94267] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 15:06:09,701 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [61f94267] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 25f373bf >2013-03-10 15:06:09,703 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [61f94267] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 15:06:09,704 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [1e9afe3e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 15:06:09,713 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-78) [157069da] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:06:09,714 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [157069da] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 133b68a6 >2013-03-10 15:06:09,715 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [157069da] FINISH, SetVdsStatusVDSCommand, log id: 133b68a6 >2013-03-10 15:06:09,721 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-78) [157069da] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 15:06:09,743 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-78) [6c084e5c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:06:09,745 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-78) [4c090c7d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:06:09,747 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-78) [4c090c7d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 15:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [e986c15] Autorecovering 1 hosts >2013-03-10 15:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [e986c15] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-3) [32a79bb2] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-3) [32a79bb2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-3) [32a79bb2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5a47d9e1 >2013-03-10 15:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-3) [32a79bb2] FINISH, SetVdsStatusVDSCommand, log id: 5a47d9e1 >2013-03-10 15:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-3) [32a79bb2] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5bc1e83 >2013-03-10 15:10:00,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-3) [32a79bb2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2a464b55 >2013-03-10 15:10:00,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-3) [32a79bb2] FINISH, GetHardwareInfoVDSCommand, log id: 2a464b55 >2013-03-10 15:10:00,247 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-3) [77048153] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:10:00,249 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-3) [18adf54] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:10:00,250 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-3) [18adf54] FINISH, ActivateVdsVDSCommand, log id: 5bc1e83 >2013-03-10 15:10:00,270 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-3) [18adf54] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:10:00,270 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) [18adf54] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 15:10:00,272 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [18adf54] Autorecovering 1 storage domains >2013-03-10 15:10:00,272 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-3) [18adf54] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 15:10:00,272 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-3) [a488127] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:10:00,273 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-3) [a488127] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 3:10 PM >2013-03-10 15:10:00,275 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-3) [a488127] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 3:10 PM >2013-03-10 15:10:00,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [6cf8518] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7d943b1e >2013-03-10 15:10:00,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [6cf8518] FINISH, GetHardwareInfoVDSCommand, log id: 7d943b1e >2013-03-10 15:10:00,985 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-52) [7e6eda07] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 15:10:00,989 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-52) [2c5ffe2d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 15:10:00,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2e694a79 >2013-03-10 15:11:08,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 15:11:08,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:11:08,973 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 15:11:08,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 2e694a79 >2013-03-10 15:11:08,977 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [633f5848] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:11:08,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [633f5848] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5d29414b >2013-03-10 15:11:08,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [633f5848] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 15:11:08,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [633f5848] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5d29414b >2013-03-10 15:11:08,993 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [633f5848] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 15:11:08,994 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [2c5ffe2d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 15:11:09,003 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-52) [4ee02d48] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:11:09,005 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [4ee02d48] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 66e347c0 >2013-03-10 15:11:09,006 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [4ee02d48] FINISH, SetVdsStatusVDSCommand, log id: 66e347c0 >2013-03-10 15:11:09,012 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-52) [4ee02d48] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 15:11:09,033 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [134cf826] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:11:09,036 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [16503809] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:11:09,037 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-52) [16503809] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 15:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [1799d88f] Autorecovering 1 hosts >2013-03-10 15:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [1799d88f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [1454e10f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [1454e10f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [1454e10f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 19f17a19 >2013-03-10 15:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [1454e10f] FINISH, SetVdsStatusVDSCommand, log id: 19f17a19 >2013-03-10 15:15:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-9) [1454e10f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 71f739c9 >2013-03-10 15:15:00,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [1454e10f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 138c5911 >2013-03-10 15:15:00,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [1454e10f] FINISH, GetHardwareInfoVDSCommand, log id: 138c5911 >2013-03-10 15:15:00,415 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-9) [55a02342] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:15:00,418 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-9) [56320f35] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:15:00,420 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-9) [56320f35] FINISH, ActivateVdsVDSCommand, log id: 71f739c9 >2013-03-10 15:15:00,441 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [56320f35] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:15:00,442 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-9) [56320f35] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 15:15:00,443 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [56320f35] Autorecovering 1 storage domains >2013-03-10 15:15:00,443 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [56320f35] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 15:15:00,444 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [2698864d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:15:00,444 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [2698864d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 3:15 PM >2013-03-10 15:15:00,447 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [2698864d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 3:15 PM >2013-03-10 15:15:00,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [68ecd198] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3bf0bde5 >2013-03-10 15:15:00,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [68ecd198] FINISH, GetHardwareInfoVDSCommand, log id: 3bf0bde5 >2013-03-10 15:15:00,520 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-16) [3ee5ae9a] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 15:15:00,524 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-16) [312cf43c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 15:15:00,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3e74d362 >2013-03-10 15:16:06,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 15:16:06,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:16:06,516 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 15:16:06,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 3e74d362 >2013-03-10 15:16:06,519 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [43f75252] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:16:06,533 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [43f75252] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2f999280 >2013-03-10 15:16:06,533 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [43f75252] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 15:16:06,533 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [43f75252] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2f999280 >2013-03-10 15:16:06,535 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [43f75252] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 15:16:06,536 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [312cf43c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 15:16:06,547 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-16) [1bc7d4b2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:16:06,548 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [1bc7d4b2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 169a8a05 >2013-03-10 15:16:06,549 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [1bc7d4b2] FINISH, SetVdsStatusVDSCommand, log id: 169a8a05 >2013-03-10 15:16:06,555 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-16) [1bc7d4b2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 15:16:06,576 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-16) [4921674f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:16:06,579 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-16) [3755b11] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:16:06,580 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-16) [3755b11] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 15:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [3f6fdad6] Autorecovering 1 hosts >2013-03-10 15:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [3f6fdad6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:20:00,027 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-67) [2d283155] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:20:00,027 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-67) [2d283155] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:20:00,032 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [2d283155] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 488b1e03 >2013-03-10 15:20:00,033 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [2d283155] FINISH, SetVdsStatusVDSCommand, log id: 488b1e03 >2013-03-10 15:20:00,035 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-67) [2d283155] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 79dc17ac >2013-03-10 15:20:00,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [2d283155] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4b28166e >2013-03-10 15:20:00,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [2d283155] FINISH, GetHardwareInfoVDSCommand, log id: 4b28166e >2013-03-10 15:20:00,409 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-67) [10c76bb9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:20:00,412 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-67) [367be853] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:20:00,413 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-67) [367be853] FINISH, ActivateVdsVDSCommand, log id: 79dc17ac >2013-03-10 15:20:00,432 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-67) [367be853] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:20:00,433 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-67) [367be853] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 15:20:00,434 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [367be853] Autorecovering 1 storage domains >2013-03-10 15:20:00,434 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [367be853] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 15:20:00,435 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-67) [546f3764] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:20:00,435 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-67) [546f3764] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 3:20 PM >2013-03-10 15:20:00,438 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-67) [546f3764] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 3:20 PM >2013-03-10 15:20:00,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [2ff1908c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 223adaa2 >2013-03-10 15:20:00,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [2ff1908c] FINISH, GetHardwareInfoVDSCommand, log id: 223adaa2 >2013-03-10 15:20:00,821 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-90) [6bfa0e62] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 15:20:00,825 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-90) [3726bb5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 15:20:00,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7c92a0f8 >2013-03-10 15:21:09,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 15:21:09,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:21:09,822 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 15:21:09,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 7c92a0f8 >2013-03-10 15:21:09,826 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [78612c00] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:21:09,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [78612c00] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 798be6fd >2013-03-10 15:21:09,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [78612c00] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 15:21:09,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [78612c00] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 798be6fd >2013-03-10 15:21:09,841 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [78612c00] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 15:21:09,842 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [3726bb5] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 15:21:09,852 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-90) [346884a1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:21:09,853 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [346884a1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 395db8c1 >2013-03-10 15:21:09,854 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [346884a1] FINISH, SetVdsStatusVDSCommand, log id: 395db8c1 >2013-03-10 15:21:09,861 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [346884a1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 15:21:09,884 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-90) [6363939c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:21:09,886 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-90) [31e5ebee] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:21:09,887 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-90) [31e5ebee] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 15:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [cf9e7f6] Autorecovering 1 hosts >2013-03-10 15:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [cf9e7f6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [132aa6d3] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [132aa6d3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [132aa6d3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 76b9c0ab >2013-03-10 15:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [132aa6d3] FINISH, SetVdsStatusVDSCommand, log id: 76b9c0ab >2013-03-10 15:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-93) [132aa6d3] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1ebbc00b >2013-03-10 15:25:00,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [132aa6d3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6fccf4e5 >2013-03-10 15:25:00,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [132aa6d3] FINISH, GetHardwareInfoVDSCommand, log id: 6fccf4e5 >2013-03-10 15:25:00,659 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [27d2d751] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:25:00,661 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [bb1540b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:25:00,693 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-93) [bb1540b] FINISH, ActivateVdsVDSCommand, log id: 1ebbc00b >2013-03-10 15:25:00,713 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [bb1540b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:25:00,714 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-93) [bb1540b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 15:25:00,715 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [bb1540b] Autorecovering 1 storage domains >2013-03-10 15:25:00,715 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [bb1540b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 15:25:00,716 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [3821fa43] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:25:00,716 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [3821fa43] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 3:25 PM >2013-03-10 15:25:00,718 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [3821fa43] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 3:25 PM >2013-03-10 15:25:01,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [66f48474] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3d311843 >2013-03-10 15:25:01,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [66f48474] FINISH, GetHardwareInfoVDSCommand, log id: 3d311843 >2013-03-10 15:25:01,126 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-50) [13d778f5] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 15:25:01,130 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-50) [8c3d6c3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 15:25:01,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 770bc4b8 >2013-03-10 15:26:04,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 15:26:04,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:26:04,132 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 15:26:04,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 770bc4b8 >2013-03-10 15:26:04,136 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [4a94cecd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:26:04,150 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4a94cecd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6e37d78a >2013-03-10 15:26:04,150 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [4a94cecd] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 15:26:04,150 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4a94cecd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6e37d78a >2013-03-10 15:26:04,152 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [4a94cecd] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 15:26:04,152 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [8c3d6c3] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 15:26:04,162 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-50) [2af5d21] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:26:04,163 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [2af5d21] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1b4718b3 >2013-03-10 15:26:04,165 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [2af5d21] FINISH, SetVdsStatusVDSCommand, log id: 1b4718b3 >2013-03-10 15:26:04,170 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-50) [2af5d21] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 15:26:04,192 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-50) [480f55f3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:26:04,195 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-50) [5a7a9a10] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:26:04,196 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-50) [5a7a9a10] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 15:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [26059eec] Autorecovering 1 hosts >2013-03-10 15:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [26059eec] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-80) [4dc699cf] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-80) [4dc699cf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [4dc699cf] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6d4d10c3 >2013-03-10 15:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [4dc699cf] FINISH, SetVdsStatusVDSCommand, log id: 6d4d10c3 >2013-03-10 15:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-80) [4dc699cf] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: e061e7c >2013-03-10 15:30:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [4dc699cf] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 438872c3 >2013-03-10 15:30:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [4dc699cf] FINISH, GetHardwareInfoVDSCommand, log id: 438872c3 >2013-03-10 15:30:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-80) [29da8ec8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:30:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-80) [549e1250] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:30:00,094 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-80) [549e1250] FINISH, ActivateVdsVDSCommand, log id: e061e7c >2013-03-10 15:30:00,113 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-80) [549e1250] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:30:00,114 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-80) [549e1250] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 15:30:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [549e1250] Autorecovering 1 storage domains >2013-03-10 15:30:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [549e1250] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 15:30:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-80) [2163b5f3] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:30:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-80) [2163b5f3] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 3:30 PM >2013-03-10 15:30:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-80) [2163b5f3] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 3:30 PM >2013-03-10 15:30:01,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [dd0195b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 60b2e533 >2013-03-10 15:30:01,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [dd0195b] FINISH, GetHardwareInfoVDSCommand, log id: 60b2e533 >2013-03-10 15:30:01,442 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-12) [f437612] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 15:30:01,445 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-12) [54d938ba] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 15:30:01,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 69fa9534 >2013-03-10 15:30:33,841 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-34) [626594b9] Start refreshing all users data >2013-03-10 15:30:34,987 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [347b8412] Attempting to update VMs/Templates Ovf. >2013-03-10 15:30:34,988 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [347b8412] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 15:30:34,989 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [347b8412] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 15:30:34,989 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [347b8412] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 15:30:34,989 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [347b8412] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 15:30:34,990 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [347b8412] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 15:30:34,990 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [347b8412] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 15:31:08,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 15:31:08,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:31:08,491 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 15:31:08,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 69fa9534 >2013-03-10 15:31:08,494 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [221dab71] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:31:08,510 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [221dab71] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2bb10e2 >2013-03-10 15:31:08,510 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [221dab71] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 15:31:08,510 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [221dab71] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2bb10e2 >2013-03-10 15:31:08,512 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [221dab71] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 15:31:08,513 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [54d938ba] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 15:31:08,527 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-12) [1eee884b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:31:08,528 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [1eee884b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5d2b5c36 >2013-03-10 15:31:08,529 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [1eee884b] FINISH, SetVdsStatusVDSCommand, log id: 5d2b5c36 >2013-03-10 15:31:08,535 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-12) [1eee884b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 15:31:08,558 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-12) [43f51b5a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:31:08,561 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-12) [1aec2c2e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:31:08,562 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-12) [1aec2c2e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 15:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [4ac01c82] Autorecovering 1 hosts >2013-03-10 15:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [4ac01c82] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:35:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [203cae83] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:35:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [203cae83] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [203cae83] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1bd70ed5 >2013-03-10 15:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [203cae83] FINISH, SetVdsStatusVDSCommand, log id: 1bd70ed5 >2013-03-10 15:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [203cae83] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1b25144a >2013-03-10 15:35:00,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [203cae83] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 50a23b8c >2013-03-10 15:35:00,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [203cae83] FINISH, GetHardwareInfoVDSCommand, log id: 50a23b8c >2013-03-10 15:35:00,398 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-73) [6b036c06] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:35:00,400 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-73) [35a59be7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:35:00,401 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [35a59be7] FINISH, ActivateVdsVDSCommand, log id: 1b25144a >2013-03-10 15:35:00,420 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [35a59be7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:35:00,421 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-73) [35a59be7] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 15:35:00,422 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [35a59be7] Autorecovering 1 storage domains >2013-03-10 15:35:00,422 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [35a59be7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 15:35:00,423 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [74f4dbd5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:35:00,423 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [74f4dbd5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 3:35 PM >2013-03-10 15:35:00,426 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [74f4dbd5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 3:35 PM >2013-03-10 15:35:02,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [3821fa43] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 665166bc >2013-03-10 15:35:02,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [3821fa43] FINISH, GetHardwareInfoVDSCommand, log id: 665166bc >2013-03-10 15:35:02,805 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-93) [42051bdd] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 15:35:02,808 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-93) [1731cd67] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 15:35:02,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 436b095 >2013-03-10 15:36:08,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 15:36:08,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:36:08,796 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 15:36:08,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 436b095 >2013-03-10 15:36:08,800 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [2ff7d344] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:36:08,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [2ff7d344] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6e09a2a3 >2013-03-10 15:36:08,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [2ff7d344] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 15:36:08,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [2ff7d344] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6e09a2a3 >2013-03-10 15:36:08,816 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [2ff7d344] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 15:36:08,817 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [1731cd67] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 15:36:08,826 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-93) [366d964a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:36:08,827 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [366d964a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 575b7c09 >2013-03-10 15:36:08,829 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [366d964a] FINISH, SetVdsStatusVDSCommand, log id: 575b7c09 >2013-03-10 15:36:08,834 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-93) [366d964a] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 15:36:08,856 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [5d3d056] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:36:08,859 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [797578a6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:36:08,860 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-93) [797578a6] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 15:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [16d33484] Autorecovering 1 hosts >2013-03-10 15:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [16d33484] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [132940bc] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [132940bc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [132940bc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 57643222 >2013-03-10 15:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [132940bc] FINISH, SetVdsStatusVDSCommand, log id: 57643222 >2013-03-10 15:40:00,037 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [132940bc] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7d13c3e1 >2013-03-10 15:40:00,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [132940bc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 710519fc >2013-03-10 15:40:00,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [132940bc] FINISH, GetHardwareInfoVDSCommand, log id: 710519fc >2013-03-10 15:40:00,363 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-14) [5d51db56] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:40:00,365 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-14) [4f1fc50f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:40:00,367 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [4f1fc50f] FINISH, ActivateVdsVDSCommand, log id: 7d13c3e1 >2013-03-10 15:40:00,386 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [4f1fc50f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:40:00,387 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-14) [4f1fc50f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 15:40:00,388 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [4f1fc50f] Autorecovering 1 storage domains >2013-03-10 15:40:00,388 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [4f1fc50f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 15:40:00,389 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [37b009bb] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:40:00,389 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [37b009bb] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 3:40 PM >2013-03-10 15:40:00,392 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [37b009bb] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 3:40 PM >2013-03-10 15:40:03,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [1213812e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 274a1a1 >2013-03-10 15:40:03,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [1213812e] FINISH, GetHardwareInfoVDSCommand, log id: 274a1a1 >2013-03-10 15:40:03,097 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-18) [439e61e6] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 15:40:03,101 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-18) [6e8624be] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 15:40:03,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5bce4774 >2013-03-10 15:41:12,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 15:41:12,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:41:12,109 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 15:41:12,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 5bce4774 >2013-03-10 15:41:12,113 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [f1876b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:41:12,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [f1876b5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 65648642 >2013-03-10 15:41:12,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [f1876b5] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 15:41:12,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [f1876b5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 65648642 >2013-03-10 15:41:12,129 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [f1876b5] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 15:41:12,130 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [6e8624be] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 15:41:12,140 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-18) [3648975d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:41:12,142 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [3648975d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 339fddda >2013-03-10 15:41:12,143 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [3648975d] FINISH, SetVdsStatusVDSCommand, log id: 339fddda >2013-03-10 15:41:12,149 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [3648975d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 15:41:12,171 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [1a26a7a2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:41:12,173 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [3f5074e0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:41:12,175 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) [3f5074e0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 15:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [2f798a3f] Autorecovering 1 hosts >2013-03-10 15:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [2f798a3f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [7f33fe93] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [7f33fe93] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [7f33fe93] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 27af4090 >2013-03-10 15:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [7f33fe93] FINISH, SetVdsStatusVDSCommand, log id: 27af4090 >2013-03-10 15:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [7f33fe93] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 20cf338c >2013-03-10 15:45:00,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [7f33fe93] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5cd347ec >2013-03-10 15:45:00,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [7f33fe93] FINISH, GetHardwareInfoVDSCommand, log id: 5cd347ec >2013-03-10 15:45:00,392 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [4bf771d1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:45:00,395 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [3aaec8a5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:45:00,396 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [3aaec8a5] FINISH, ActivateVdsVDSCommand, log id: 20cf338c >2013-03-10 15:45:00,416 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [3aaec8a5] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:45:00,417 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [3aaec8a5] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 15:45:00,418 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [3aaec8a5] Autorecovering 1 storage domains >2013-03-10 15:45:00,418 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [3aaec8a5] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 15:45:00,419 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [3b5c047] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:45:00,419 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [3b5c047] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 3:45 PM >2013-03-10 15:45:00,422 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [3b5c047] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 3:45 PM >2013-03-10 15:45:00,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [77649e34] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6c5b9bac >2013-03-10 15:45:00,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [77649e34] FINISH, GetHardwareInfoVDSCommand, log id: 6c5b9bac >2013-03-10 15:45:00,496 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-57) [6b09d31c] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 15:45:00,499 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-57) [3facaac5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 15:45:00,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 64c6afd9 >2013-03-10 15:46:06,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 15:46:06,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:46:06,464 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 15:46:06,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 64c6afd9 >2013-03-10 15:46:06,468 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [18ac3395] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:46:06,492 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [18ac3395] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 13f50ee7 >2013-03-10 15:46:06,492 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [18ac3395] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 15:46:06,492 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [18ac3395] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 13f50ee7 >2013-03-10 15:46:06,495 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [18ac3395] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 15:46:06,495 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [3facaac5] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 15:46:06,508 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-57) [1db807f6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:46:06,510 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [1db807f6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 43a449e5 >2013-03-10 15:46:06,511 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [1db807f6] FINISH, SetVdsStatusVDSCommand, log id: 43a449e5 >2013-03-10 15:46:06,518 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-57) [1db807f6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 15:46:06,544 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-57) [457c834b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:46:06,547 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-57) [6e9b1af2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:46:06,548 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-57) [6e9b1af2] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 15:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [4379195e] Autorecovering 1 hosts >2013-03-10 15:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [4379195e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [3fadf8a3] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [3fadf8a3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [3fadf8a3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7bba3acd >2013-03-10 15:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [3fadf8a3] FINISH, SetVdsStatusVDSCommand, log id: 7bba3acd >2013-03-10 15:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [3fadf8a3] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 51fb3253 >2013-03-10 15:50:00,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [3fadf8a3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2ba7b798 >2013-03-10 15:50:00,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [3fadf8a3] FINISH, GetHardwareInfoVDSCommand, log id: 2ba7b798 >2013-03-10 15:50:00,423 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-69) [7a3a01d3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:50:00,426 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-69) [4c302929] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:50:00,428 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [4c302929] FINISH, ActivateVdsVDSCommand, log id: 51fb3253 >2013-03-10 15:50:00,447 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [4c302929] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:50:00,448 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-69) [4c302929] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 15:50:00,449 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [4c302929] Autorecovering 1 storage domains >2013-03-10 15:50:00,449 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [4c302929] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 15:50:00,450 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [18811a0a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:50:00,450 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [18811a0a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 3:50 PM >2013-03-10 15:50:00,453 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [18811a0a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 3:50 PM >2013-03-10 15:50:00,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [6040c924] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4a63119b >2013-03-10 15:50:00,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [6040c924] FINISH, GetHardwareInfoVDSCommand, log id: 4a63119b >2013-03-10 15:50:00,800 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-22) [1e49666b] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 15:50:00,804 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-22) [4255f734] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 15:50:00,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: e1fe42a >2013-03-10 15:51:07,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 15:51:07,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:51:07,799 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 15:51:07,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: e1fe42a >2013-03-10 15:51:07,803 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [552e7ebd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:51:07,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [552e7ebd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 42b41f55 >2013-03-10 15:51:07,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [552e7ebd] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 15:51:07,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [552e7ebd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 42b41f55 >2013-03-10 15:51:07,819 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [552e7ebd] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 15:51:07,820 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [4255f734] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 15:51:07,829 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-22) [2c162d42] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:51:07,831 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [2c162d42] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4093df5c >2013-03-10 15:51:07,832 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [2c162d42] FINISH, SetVdsStatusVDSCommand, log id: 4093df5c >2013-03-10 15:51:07,838 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [2c162d42] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 15:51:07,863 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [14e9a68c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:51:07,865 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [cc5193e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:51:07,866 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-22) [cc5193e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 15:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [347b8412] Autorecovering 1 hosts >2013-03-10 15:55:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [347b8412] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [6c0506e3] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:55:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [6c0506e3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [6c0506e3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 57f5ea07 >2013-03-10 15:55:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [6c0506e3] FINISH, SetVdsStatusVDSCommand, log id: 57f5ea07 >2013-03-10 15:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [6c0506e3] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 25ae2746 >2013-03-10 15:55:00,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [6c0506e3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3c7a14ac >2013-03-10 15:55:00,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [6c0506e3] FINISH, GetHardwareInfoVDSCommand, log id: 3c7a14ac >2013-03-10 15:55:00,223 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [4aaf8969] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:55:00,225 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [18d94a6f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:55:00,226 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [18d94a6f] FINISH, ActivateVdsVDSCommand, log id: 25ae2746 >2013-03-10 15:55:00,246 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [18d94a6f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 15:55:00,247 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [18d94a6f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 15:55:00,248 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [18d94a6f] Autorecovering 1 storage domains >2013-03-10 15:55:00,248 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [18d94a6f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 15:55:00,249 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [1348b24b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:55:00,249 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [1348b24b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 3:55 PM >2013-03-10 15:55:00,251 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [1348b24b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 3:55 PM >2013-03-10 15:55:02,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [6887909e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6caf05a0 >2013-03-10 15:55:02,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [6887909e] FINISH, GetHardwareInfoVDSCommand, log id: 6caf05a0 >2013-03-10 15:55:02,112 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-84) [24b09b10] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 15:55:02,116 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-84) [2e9a9fe9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 15:55:02,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 62bc583b >2013-03-10 15:56:11,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 15:56:11,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 15:56:11,197 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 15:56:11,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 62bc583b >2013-03-10 15:56:11,202 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [714985c2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 15:56:11,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [714985c2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 749ce6e4 >2013-03-10 15:56:11,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [714985c2] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 15:56:11,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [714985c2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 749ce6e4 >2013-03-10 15:56:11,233 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [2e9a9fe9] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 15:56:11,242 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-84) [3b96361c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:56:11,244 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [3b96361c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 303c5236 >2013-03-10 15:56:11,245 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [3b96361c] FINISH, SetVdsStatusVDSCommand, log id: 303c5236 >2013-03-10 15:56:11,251 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-84) [3b96361c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 15:56:11,276 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-84) [71b8ca75] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:56:11,278 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-84) [38ad567d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 15:56:11,280 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-84) [38ad567d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 16:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [45d010ba] Autorecovering 1 hosts >2013-03-10 16:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [45d010ba] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [7eadb156] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [7eadb156] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [7eadb156] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6e3647d1 >2013-03-10 16:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [7eadb156] FINISH, SetVdsStatusVDSCommand, log id: 6e3647d1 >2013-03-10 16:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-87) [7eadb156] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2a39ee16 >2013-03-10 16:00:00,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [7eadb156] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 63c6787d >2013-03-10 16:00:00,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [7eadb156] FINISH, GetHardwareInfoVDSCommand, log id: 63c6787d >2013-03-10 16:00:00,230 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [47957f72] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:00:00,232 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [30cdbac] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:00:00,234 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-87) [30cdbac] FINISH, ActivateVdsVDSCommand, log id: 2a39ee16 >2013-03-10 16:00:00,253 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [30cdbac] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:00:00,254 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [30cdbac] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 16:00:00,255 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [30cdbac] Autorecovering 1 storage domains >2013-03-10 16:00:00,255 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [30cdbac] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 16:00:00,256 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [5be3bae] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:00:00,256 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [5be3bae] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 4:00 PM >2013-03-10 16:00:00,259 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [5be3bae] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 4:00 PM >2013-03-10 16:00:02,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [16c58dbf] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3fb42cf >2013-03-10 16:00:02,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [16c58dbf] FINISH, GetHardwareInfoVDSCommand, log id: 3fb42cf >2013-03-10 16:00:02,529 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-81) [72db1dd2] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 16:00:02,532 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-81) [2b699912] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 16:00:02,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5fc4228d >2013-03-10 16:01:07,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 16:01:07,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:01:07,568 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 16:01:07,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 5fc4228d >2013-03-10 16:01:07,572 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [9ea71fb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:01:07,587 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [9ea71fb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3907755f >2013-03-10 16:01:07,587 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [9ea71fb] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 16:01:07,587 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [9ea71fb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3907755f >2013-03-10 16:01:07,589 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [9ea71fb] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 16:01:07,589 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [2b699912] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 16:01:07,599 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-81) [63914c5b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:01:07,600 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [63914c5b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4dccea3f >2013-03-10 16:01:07,601 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [63914c5b] FINISH, SetVdsStatusVDSCommand, log id: 4dccea3f >2013-03-10 16:01:07,607 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-81) [63914c5b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 16:01:07,646 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-81) [2ea134ce] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:01:07,649 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-81) [5cb70b17] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:01:07,650 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-81) [5cb70b17] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 16:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [2bb2d7fd] Autorecovering 1 hosts >2013-03-10 16:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [2bb2d7fd] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [61c450ad] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [61c450ad] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-29) [61c450ad] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 44b89754 >2013-03-10 16:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-29) [61c450ad] FINISH, SetVdsStatusVDSCommand, log id: 44b89754 >2013-03-10 16:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-29) [61c450ad] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 65fd0d2 >2013-03-10 16:05:00,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-29) [61c450ad] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2520d3b0 >2013-03-10 16:05:00,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-29) [61c450ad] FINISH, GetHardwareInfoVDSCommand, log id: 2520d3b0 >2013-03-10 16:05:00,376 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-29) [51dc88a2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:05:00,378 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-29) [109b6100] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:05:00,380 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-29) [109b6100] FINISH, ActivateVdsVDSCommand, log id: 65fd0d2 >2013-03-10 16:05:00,403 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [109b6100] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:05:00,404 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-29) [109b6100] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 16:05:00,405 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [109b6100] Autorecovering 1 storage domains >2013-03-10 16:05:00,405 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [109b6100] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 16:05:00,406 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [46af7501] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:05:00,406 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [46af7501] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 4:05 PM >2013-03-10 16:05:00,409 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [46af7501] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 4:05 PM >2013-03-10 16:05:01,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [4c5ace3e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 38a70d4b >2013-03-10 16:05:01,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [4c5ace3e] FINISH, GetHardwareInfoVDSCommand, log id: 38a70d4b >2013-03-10 16:05:01,900 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-95) [7ddcea4b] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 16:05:01,904 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-95) [a04457f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 16:05:01,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2d09bfc0 >2013-03-10 16:06:08,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 16:06:08,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:06:08,938 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 16:06:08,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 2d09bfc0 >2013-03-10 16:06:08,942 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [6cf4c66c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:06:08,965 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [6cf4c66c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 473abfb7 >2013-03-10 16:06:08,965 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [6cf4c66c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 16:06:08,965 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [6cf4c66c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 473abfb7 >2013-03-10 16:06:08,967 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [6cf4c66c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 16:06:08,967 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [a04457f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 16:06:08,977 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-95) [70c16ff4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:06:08,979 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [70c16ff4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5029224f >2013-03-10 16:06:08,980 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [70c16ff4] FINISH, SetVdsStatusVDSCommand, log id: 5029224f >2013-03-10 16:06:08,985 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-95) [70c16ff4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 16:06:09,007 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-95) [75154424] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:06:09,010 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-95) [45627a46] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:06:09,011 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-95) [45627a46] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 16:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [6566b16c] Autorecovering 1 hosts >2013-03-10 16:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [6566b16c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-10) [201f9da4] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-10) [201f9da4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [201f9da4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3aa11c50 >2013-03-10 16:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [201f9da4] FINISH, SetVdsStatusVDSCommand, log id: 3aa11c50 >2013-03-10 16:10:00,025 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-10) [201f9da4] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1ae25e5b >2013-03-10 16:10:02,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [201f9da4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 31cf63a7 >2013-03-10 16:10:02,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [201f9da4] FINISH, GetHardwareInfoVDSCommand, log id: 31cf63a7 >2013-03-10 16:10:02,222 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-10) [43728fbe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:10:02,225 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-10) [4a87c98a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:10:02,227 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-10) [4a87c98a] FINISH, ActivateVdsVDSCommand, log id: 1ae25e5b >2013-03-10 16:10:02,246 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-10) [4a87c98a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:10:02,247 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-10) [4a87c98a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 16:10:02,248 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [4a87c98a] Autorecovering 1 storage domains >2013-03-10 16:10:02,248 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-10) [4a87c98a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 16:10:02,249 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-10) [7bf8535c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:10:02,249 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-10) [7bf8535c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 4:10 PM >2013-03-10 16:10:02,252 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-10) [7bf8535c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 4:10 PM >2013-03-10 16:10:02,267 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [4b4a3191] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 45034d1e >2013-03-10 16:10:02,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [4b4a3191] FINISH, GetHardwareInfoVDSCommand, log id: 45034d1e >2013-03-10 16:10:02,327 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-86) [69939644] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 16:10:02,330 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-86) [64ec71ef] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 16:10:02,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7ed48180 >2013-03-10 16:11:08,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 16:11:08,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:11:08,331 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 16:11:08,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 7ed48180 >2013-03-10 16:11:08,334 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [67ddfe27] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:11:08,348 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [67ddfe27] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 281b2327 >2013-03-10 16:11:08,348 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [67ddfe27] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 16:11:08,348 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [67ddfe27] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 281b2327 >2013-03-10 16:11:08,350 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [67ddfe27] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 16:11:08,351 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [64ec71ef] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 16:11:08,364 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-86) [648da1ff] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:11:08,366 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [648da1ff] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 577f6584 >2013-03-10 16:11:08,367 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [648da1ff] FINISH, SetVdsStatusVDSCommand, log id: 577f6584 >2013-03-10 16:11:08,372 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-86) [648da1ff] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 16:11:08,393 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-86) [2b0a37c3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:11:08,396 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-86) [68c3b638] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:11:08,397 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-86) [68c3b638] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 16:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [722b2686] Autorecovering 1 hosts >2013-03-10 16:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [722b2686] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [3c1dba30] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [3c1dba30] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [3c1dba30] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3c289d8e >2013-03-10 16:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [3c1dba30] FINISH, SetVdsStatusVDSCommand, log id: 3c289d8e >2013-03-10 16:15:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-21) [3c1dba30] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5db0e9e1 >2013-03-10 16:15:00,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [3c1dba30] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1b488216 >2013-03-10 16:15:00,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [3c1dba30] FINISH, GetHardwareInfoVDSCommand, log id: 1b488216 >2013-03-10 16:15:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-21) [f818db0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:15:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-21) [3085c829] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:15:00,094 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-21) [3085c829] FINISH, ActivateVdsVDSCommand, log id: 5db0e9e1 >2013-03-10 16:15:00,113 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [3085c829] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:15:00,114 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-21) [3085c829] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 16:15:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [3085c829] Autorecovering 1 storage domains >2013-03-10 16:15:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [3085c829] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 16:15:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [6675721] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:15:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [6675721] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 4:15 PM >2013-03-10 16:15:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [6675721] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 4:15 PM >2013-03-10 16:15:02,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [3b0749de] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4a5f3ce3 >2013-03-10 16:15:02,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [3b0749de] FINISH, GetHardwareInfoVDSCommand, log id: 4a5f3ce3 >2013-03-10 16:15:02,644 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-24) [34d16905] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 16:15:02,647 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-24) [648e2db1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 16:15:02,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6e9864bb >2013-03-10 16:16:09,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 16:16:09,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:16:09,655 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 16:16:09,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 6e9864bb >2013-03-10 16:16:09,659 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [befd448] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:16:09,672 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [befd448] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: eee6aab >2013-03-10 16:16:09,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [befd448] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 16:16:09,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [befd448] FINISH, MarkPoolInReconstructModeVDSCommand, log id: eee6aab >2013-03-10 16:16:09,675 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [befd448] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 16:16:09,675 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [648e2db1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 16:16:09,685 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-24) [10e20e8c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:16:09,686 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [10e20e8c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5dc12ac >2013-03-10 16:16:09,687 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [10e20e8c] FINISH, SetVdsStatusVDSCommand, log id: 5dc12ac >2013-03-10 16:16:09,693 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-24) [10e20e8c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 16:16:09,714 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-24) [109339a8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:16:09,716 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-24) [6687b8b8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:16:09,718 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-24) [6687b8b8] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 16:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [2fb329b2] Autorecovering 1 hosts >2013-03-10 16:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [2fb329b2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [75de6b43] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [75de6b43] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [75de6b43] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 69f53ef0 >2013-03-10 16:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [75de6b43] FINISH, SetVdsStatusVDSCommand, log id: 69f53ef0 >2013-03-10 16:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-89) [75de6b43] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1da440b >2013-03-10 16:20:00,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [75de6b43] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 312a1435 >2013-03-10 16:20:00,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [75de6b43] FINISH, GetHardwareInfoVDSCommand, log id: 312a1435 >2013-03-10 16:20:00,455 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-89) [21f429ba] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:20:00,457 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-89) [99cc622] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:20:00,459 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-89) [99cc622] FINISH, ActivateVdsVDSCommand, log id: 1da440b >2013-03-10 16:20:00,478 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [99cc622] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:20:00,479 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-89) [99cc622] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 16:20:00,480 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [99cc622] Autorecovering 1 storage domains >2013-03-10 16:20:00,480 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [99cc622] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 16:20:00,481 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [7e74c320] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:20:00,481 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [7e74c320] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 4:20 PM >2013-03-10 16:20:00,484 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [7e74c320] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 4:20 PM >2013-03-10 16:20:00,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [3f5074e0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2a4c30db >2013-03-10 16:20:00,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [3f5074e0] FINISH, GetHardwareInfoVDSCommand, log id: 2a4c30db >2013-03-10 16:20:00,961 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-18) [663fe98c] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 16:20:00,964 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-18) [15a31c39] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 16:20:00,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 803e9e >2013-03-10 16:21:07,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 16:21:07,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:21:07,965 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 16:21:07,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 803e9e >2013-03-10 16:21:07,968 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [56fb5ba6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:21:07,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [56fb5ba6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 22d6ef5d >2013-03-10 16:21:07,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [56fb5ba6] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 16:21:07,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [56fb5ba6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 22d6ef5d >2013-03-10 16:21:07,986 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [56fb5ba6] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 16:21:07,986 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [15a31c39] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 16:21:07,996 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-18) [6df6ba5c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:21:07,998 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [6df6ba5c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4776e14b >2013-03-10 16:21:07,999 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [6df6ba5c] FINISH, SetVdsStatusVDSCommand, log id: 4776e14b >2013-03-10 16:21:08,004 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [6df6ba5c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 16:21:08,027 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [29af1117] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:21:08,030 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [76b925b4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:21:08,031 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) [76b925b4] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 16:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [739675ba] Autorecovering 1 hosts >2013-03-10 16:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [739675ba] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-56) [6ae55807] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-56) [6ae55807] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [6ae55807] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 42669646 >2013-03-10 16:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [6ae55807] FINISH, SetVdsStatusVDSCommand, log id: 42669646 >2013-03-10 16:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-56) [6ae55807] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3ff6df0c >2013-03-10 16:25:00,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [6ae55807] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 19a9a460 >2013-03-10 16:25:00,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [6ae55807] FINISH, GetHardwareInfoVDSCommand, log id: 19a9a460 >2013-03-10 16:25:00,087 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-56) [5721320a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:25:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-56) [2a1e17c2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:25:00,091 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-56) [2a1e17c2] FINISH, ActivateVdsVDSCommand, log id: 3ff6df0c >2013-03-10 16:25:00,110 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-56) [2a1e17c2] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:25:00,111 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-56) [2a1e17c2] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 16:25:00,112 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [2a1e17c2] Autorecovering 1 storage domains >2013-03-10 16:25:00,112 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [2a1e17c2] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 16:25:00,113 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-56) [29f9f71f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:25:00,113 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-56) [29f9f71f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 4:25 PM >2013-03-10 16:25:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-56) [29f9f71f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 4:25 PM >2013-03-10 16:25:02,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [5a7a9a10] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 26ec99df >2013-03-10 16:25:02,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [5a7a9a10] FINISH, GetHardwareInfoVDSCommand, log id: 26ec99df >2013-03-10 16:25:02,280 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-50) [2e3e946b] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 16:25:02,284 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-50) [6546ea8a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 16:25:02,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3b756ef6 >2013-03-10 16:26:10,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 16:26:10,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:26:10,298 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 16:26:10,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 3b756ef6 >2013-03-10 16:26:10,302 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [24676e2c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:26:10,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [24676e2c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 23b7f76b >2013-03-10 16:26:10,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [24676e2c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 16:26:10,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [24676e2c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 23b7f76b >2013-03-10 16:26:10,318 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [24676e2c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 16:26:10,318 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [6546ea8a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 16:26:10,328 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-50) [22c82b3e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:26:10,329 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [22c82b3e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 65a1a08a >2013-03-10 16:26:10,330 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [22c82b3e] FINISH, SetVdsStatusVDSCommand, log id: 65a1a08a >2013-03-10 16:26:10,336 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-50) [22c82b3e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 16:26:10,357 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-50) [77c3452d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:26:10,360 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-50) [167e1326] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:26:10,361 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-50) [167e1326] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 16:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [4cc072b] Autorecovering 1 hosts >2013-03-10 16:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [4cc072b] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [52f95e7f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [52f95e7f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [52f95e7f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 48ab8f33 >2013-03-10 16:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [52f95e7f] FINISH, SetVdsStatusVDSCommand, log id: 48ab8f33 >2013-03-10 16:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-32) [52f95e7f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2eadfb3f >2013-03-10 16:30:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [52f95e7f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: ce83000 >2013-03-10 16:30:00,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [52f95e7f] FINISH, GetHardwareInfoVDSCommand, log id: ce83000 >2013-03-10 16:30:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-32) [41975478] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:30:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-32) [62f4942f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:30:00,095 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-32) [62f4942f] FINISH, ActivateVdsVDSCommand, log id: 2eadfb3f >2013-03-10 16:30:00,115 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [62f4942f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:30:00,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-32) [62f4942f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 16:30:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [62f4942f] Autorecovering 1 storage domains >2013-03-10 16:30:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [62f4942f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 16:30:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [602f56c8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:30:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [602f56c8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 4:30 PM >2013-03-10 16:30:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [602f56c8] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 4:30 PM >2013-03-10 16:30:01,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [1e3fe091] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 649e6894 >2013-03-10 16:30:01,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [1e3fe091] FINISH, GetHardwareInfoVDSCommand, log id: 649e6894 >2013-03-10 16:30:01,651 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-51) [6e0f690b] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 16:30:01,655 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-51) [637719b5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 16:30:01,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 96498a8 >2013-03-10 16:30:33,843 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-38) [5d712191] Start refreshing all users data >2013-03-10 16:30:34,991 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-35) [3b5a8bb6] Attempting to update VMs/Templates Ovf. >2013-03-10 16:30:34,992 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-35) [3b5a8bb6] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 16:30:34,992 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-35) [3b5a8bb6] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 16:30:34,993 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-35) [3b5a8bb6] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 16:30:34,993 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-35) [3b5a8bb6] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 16:30:34,993 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-35) [3b5a8bb6] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 16:30:34,994 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-35) [3b5a8bb6] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 16:31:08,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 16:31:08,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:31:08,695 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 16:31:08,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 96498a8 >2013-03-10 16:31:08,699 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [3ded6a95] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:31:08,714 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [3ded6a95] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 61c7983a >2013-03-10 16:31:08,714 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [3ded6a95] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 16:31:08,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [3ded6a95] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 61c7983a >2013-03-10 16:31:08,717 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [3ded6a95] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 16:31:08,717 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [637719b5] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 16:31:08,727 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-51) [484ceba8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:31:08,729 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [484ceba8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5cbf7615 >2013-03-10 16:31:08,730 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [484ceba8] FINISH, SetVdsStatusVDSCommand, log id: 5cbf7615 >2013-03-10 16:31:08,737 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-51) [484ceba8] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 16:31:08,759 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-51) [449390ee] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:31:08,761 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-51) [35f983d2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:31:08,763 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-51) [35f983d2] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 16:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [4d768856] Autorecovering 1 hosts >2013-03-10 16:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [4d768856] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [74d3814c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [74d3814c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [74d3814c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 77bf632e >2013-03-10 16:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [74d3814c] FINISH, SetVdsStatusVDSCommand, log id: 77bf632e >2013-03-10 16:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-92) [74d3814c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 9a78de8 >2013-03-10 16:35:00,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [74d3814c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 41d7c9cc >2013-03-10 16:35:00,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [74d3814c] FINISH, GetHardwareInfoVDSCommand, log id: 41d7c9cc >2013-03-10 16:35:00,479 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-92) [6ab9ce15] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:35:00,481 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-92) [4328085d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:35:00,483 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-92) [4328085d] FINISH, ActivateVdsVDSCommand, log id: 9a78de8 >2013-03-10 16:35:00,502 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [4328085d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:35:00,503 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-92) [4328085d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 16:35:00,504 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [4328085d] Autorecovering 1 storage domains >2013-03-10 16:35:00,504 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [4328085d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 16:35:00,505 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [56ef438c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:35:00,505 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [56ef438c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 4:35 PM >2013-03-10 16:35:00,508 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [56ef438c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 4:35 PM >2013-03-10 16:35:02,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [75083f49] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1af71616 >2013-03-10 16:35:02,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [75083f49] FINISH, GetHardwareInfoVDSCommand, log id: 1af71616 >2013-03-10 16:35:03,031 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-74) [6d374d95] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 16:35:03,035 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-74) [74e0a246] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 16:35:03,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1408a5a8 >2013-03-10 16:36:09,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 16:36:09,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:36:09,024 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 16:36:09,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 1408a5a8 >2013-03-10 16:36:09,028 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [71c4a91] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:36:09,042 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [71c4a91] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7d33edd0 >2013-03-10 16:36:09,042 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [71c4a91] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 16:36:09,042 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [71c4a91] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7d33edd0 >2013-03-10 16:36:09,044 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [71c4a91] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 16:36:09,045 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [74e0a246] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 16:36:09,055 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-74) [3608e68] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:36:09,056 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [3608e68] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6daa62b0 >2013-03-10 16:36:09,057 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [3608e68] FINISH, SetVdsStatusVDSCommand, log id: 6daa62b0 >2013-03-10 16:36:09,063 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-74) [3608e68] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 16:36:09,086 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-74) [5c79755] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:36:09,088 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-74) [6cfef8f0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:36:09,089 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-74) [6cfef8f0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 16:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [74f4dbd5] Autorecovering 1 hosts >2013-03-10 16:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [74f4dbd5] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [1db4b76f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [1db4b76f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [1db4b76f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6afbe003 >2013-03-10 16:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [1db4b76f] FINISH, SetVdsStatusVDSCommand, log id: 6afbe003 >2013-03-10 16:40:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [1db4b76f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2d0e22af >2013-03-10 16:40:00,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [1db4b76f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 644d06f9 >2013-03-10 16:40:00,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [1db4b76f] FINISH, GetHardwareInfoVDSCommand, log id: 644d06f9 >2013-03-10 16:40:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-73) [3ac05c1b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:40:00,095 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-73) [10e13a26] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:40:00,096 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [10e13a26] FINISH, ActivateVdsVDSCommand, log id: 2d0e22af >2013-03-10 16:40:00,126 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [10e13a26] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:40:00,127 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-73) [10e13a26] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 16:40:00,129 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [10e13a26] Autorecovering 1 storage domains >2013-03-10 16:40:00,129 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [10e13a26] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 16:40:00,129 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [3803c270] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:40:00,130 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [3803c270] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 4:40 PM >2013-03-10 16:40:00,132 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [3803c270] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 4:40 PM >2013-03-10 16:40:00,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [316b7163] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 47e75712 >2013-03-10 16:40:00,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [316b7163] FINISH, GetHardwareInfoVDSCommand, log id: 47e75712 >2013-03-10 16:40:00,335 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-100) [1cc6a2c9] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 16:40:00,339 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-100) [5aab787e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 16:40:00,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5a6f3c90 >2013-03-10 16:41:07,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 16:41:07,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:41:07,350 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 16:41:07,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 5a6f3c90 >2013-03-10 16:41:07,354 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [a4e8f94] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:41:07,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [a4e8f94] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: df5aeda >2013-03-10 16:41:07,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [a4e8f94] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 16:41:07,368 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [a4e8f94] FINISH, MarkPoolInReconstructModeVDSCommand, log id: df5aeda >2013-03-10 16:41:07,370 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [a4e8f94] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 16:41:07,370 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [5aab787e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 16:41:07,380 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-100) [41ab4f7d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:41:07,382 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [41ab4f7d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4e4faf3 >2013-03-10 16:41:07,383 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [41ab4f7d] FINISH, SetVdsStatusVDSCommand, log id: 4e4faf3 >2013-03-10 16:41:07,388 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-100) [41ab4f7d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 16:41:07,413 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-100) [5e6ca12d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:41:07,416 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-100) [10ce9d9a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:41:07,417 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-100) [10ce9d9a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 16:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [58449771] Autorecovering 1 hosts >2013-03-10 16:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [58449771] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [2e836db0] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [2e836db0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:45:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [2e836db0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: a2ce2b6 >2013-03-10 16:45:00,017 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [2e836db0] FINISH, SetVdsStatusVDSCommand, log id: a2ce2b6 >2013-03-10 16:45:00,020 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-46) [2e836db0] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2a49c96d >2013-03-10 16:45:00,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [2e836db0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 668e8a1c >2013-03-10 16:45:00,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [2e836db0] FINISH, GetHardwareInfoVDSCommand, log id: 668e8a1c >2013-03-10 16:45:00,214 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-46) [3c76b7e4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:45:00,216 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-46) [540a10bc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:45:00,218 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-46) [540a10bc] FINISH, ActivateVdsVDSCommand, log id: 2a49c96d >2013-03-10 16:45:00,238 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [540a10bc] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:45:00,239 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-46) [540a10bc] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 16:45:00,240 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [540a10bc] Autorecovering 1 storage domains >2013-03-10 16:45:00,240 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [540a10bc] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 16:45:00,241 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [326dbb02] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:45:00,241 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [326dbb02] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 4:45 PM >2013-03-10 16:45:00,244 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [326dbb02] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 4:45 PM >2013-03-10 16:45:01,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [2163b5f3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4e342a24 >2013-03-10 16:45:01,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [2163b5f3] FINISH, GetHardwareInfoVDSCommand, log id: 4e342a24 >2013-03-10 16:45:01,665 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-80) [3e2a67af] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 16:45:01,668 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-80) [39ee8598] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 16:45:01,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2e97ae1e >2013-03-10 16:46:10,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 16:46:10,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:46:10,667 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 16:46:10,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 2e97ae1e >2013-03-10 16:46:10,670 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [4dc66ce4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:46:10,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [4dc66ce4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 10ca8b8e >2013-03-10 16:46:10,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [4dc66ce4] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 16:46:10,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [4dc66ce4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 10ca8b8e >2013-03-10 16:46:10,688 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [4dc66ce4] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 16:46:10,689 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [39ee8598] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 16:46:10,710 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-80) [1fc32712] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:46:10,711 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [1fc32712] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 684d2f6d >2013-03-10 16:46:10,713 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [1fc32712] FINISH, SetVdsStatusVDSCommand, log id: 684d2f6d >2013-03-10 16:46:10,718 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-80) [1fc32712] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 16:46:10,751 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-80) [32266881] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:46:10,754 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-80) [35f834a6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:46:10,761 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-80) [35f834a6] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 16:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [7efe441c] Autorecovering 1 hosts >2013-03-10 16:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [7efe441c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-15) [46518795] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-15) [46518795] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [46518795] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1ce948ad >2013-03-10 16:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [46518795] FINISH, SetVdsStatusVDSCommand, log id: 1ce948ad >2013-03-10 16:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-15) [46518795] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2c29c920 >2013-03-10 16:50:00,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [46518795] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 36e33c6c >2013-03-10 16:50:00,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [46518795] FINISH, GetHardwareInfoVDSCommand, log id: 36e33c6c >2013-03-10 16:50:00,187 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-15) [74b0a6ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:50:00,189 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-15) [5e1a673] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:50:00,190 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-15) [5e1a673] FINISH, ActivateVdsVDSCommand, log id: 2c29c920 >2013-03-10 16:50:00,210 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-15) [5e1a673] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:50:00,211 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-15) [5e1a673] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 16:50:00,213 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [5e1a673] Autorecovering 1 storage domains >2013-03-10 16:50:00,213 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [5e1a673] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 16:50:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-15) [7750554c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:50:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-15) [7750554c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 4:50 PM >2013-03-10 16:50:00,216 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-15) [7750554c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 4:50 PM >2013-03-10 16:50:01,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [5cd7152c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2ac677ab >2013-03-10 16:50:01,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [5cd7152c] FINISH, GetHardwareInfoVDSCommand, log id: 2ac677ab >2013-03-10 16:50:02,008 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-99) [3941e5dc] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 16:50:02,011 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-99) [e8e32e3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 16:50:02,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 42f63886 >2013-03-10 16:51:10,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 16:51:10,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:51:10,019 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 16:51:10,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 42f63886 >2013-03-10 16:51:10,023 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [45aee1c4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:51:10,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [45aee1c4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 230a8c87 >2013-03-10 16:51:10,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [45aee1c4] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 16:51:10,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [45aee1c4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 230a8c87 >2013-03-10 16:51:10,039 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [45aee1c4] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 16:51:10,039 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [e8e32e3] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 16:51:10,049 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-99) [6bc8ae06] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:51:10,050 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [6bc8ae06] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7df76471 >2013-03-10 16:51:10,051 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [6bc8ae06] FINISH, SetVdsStatusVDSCommand, log id: 7df76471 >2013-03-10 16:51:10,059 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-99) [6bc8ae06] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 16:51:10,080 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-99) [69ef1129] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:51:10,083 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-99) [6eef8afb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:51:10,084 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-99) [6eef8afb] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 16:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [3b5a8bb6] Autorecovering 1 hosts >2013-03-10 16:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [3b5a8bb6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [6e993913] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [6e993913] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [6e993913] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1b6ba636 >2013-03-10 16:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [6e993913] FINISH, SetVdsStatusVDSCommand, log id: 1b6ba636 >2013-03-10 16:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-35) [6e993913] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 555023a >2013-03-10 16:55:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [6e993913] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 603e573c >2013-03-10 16:55:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [6e993913] FINISH, GetHardwareInfoVDSCommand, log id: 603e573c >2013-03-10 16:55:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-35) [65d49532] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:55:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-35) [2efda767] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:55:00,203 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-35) [2efda767] FINISH, ActivateVdsVDSCommand, log id: 555023a >2013-03-10 16:55:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [2efda767] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 16:55:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-35) [2efda767] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 16:55:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [2efda767] Autorecovering 1 storage domains >2013-03-10 16:55:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [2efda767] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 16:55:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [39683fa4] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:55:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [39683fa4] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 4:55 PM >2013-03-10 16:55:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [39683fa4] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 4:55 PM >2013-03-10 16:55:01,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [17fd5d6f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7afa6eeb >2013-03-10 16:55:01,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [17fd5d6f] FINISH, GetHardwareInfoVDSCommand, log id: 7afa6eeb >2013-03-10 16:55:01,321 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-41) [6435c4ca] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 16:55:01,325 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-41) [237183c6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 16:55:01,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 32f52a39 >2013-03-10 16:56:05,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 16:56:05,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 16:56:05,645 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 16:56:05,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 32f52a39 >2013-03-10 16:56:05,649 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [ae529fd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 16:56:05,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [ae529fd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 117df73a >2013-03-10 16:56:05,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [ae529fd] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 16:56:05,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [ae529fd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 117df73a >2013-03-10 16:56:05,665 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [ae529fd] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 16:56:05,665 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [237183c6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 16:56:05,675 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-41) [1ac301ca] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:56:05,677 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [1ac301ca] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 10eba8d0 >2013-03-10 16:56:05,678 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [1ac301ca] FINISH, SetVdsStatusVDSCommand, log id: 10eba8d0 >2013-03-10 16:56:05,715 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-41) [1ac301ca] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 16:56:05,739 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-41) [153b8dc1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:56:05,741 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-41) [40c4fe46] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 16:56:05,743 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-41) [40c4fe46] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 17:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [2c4ffb2] Autorecovering 1 hosts >2013-03-10 17:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [2c4ffb2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [4639f68c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [4639f68c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [4639f68c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 49f18524 >2013-03-10 17:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [4639f68c] FINISH, SetVdsStatusVDSCommand, log id: 49f18524 >2013-03-10 17:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-88) [4639f68c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1695a16b >2013-03-10 17:00:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [4639f68c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 294c17b8 >2013-03-10 17:00:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [4639f68c] FINISH, GetHardwareInfoVDSCommand, log id: 294c17b8 >2013-03-10 17:00:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-88) [6594bc24] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:00:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-88) [6b4ea08a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:00:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-88) [6b4ea08a] FINISH, ActivateVdsVDSCommand, log id: 1695a16b >2013-03-10 17:00:00,222 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [6b4ea08a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:00:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [6b4ea08a] Autorecovering 1 storage domains >2013-03-10 17:00:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [6b4ea08a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 17:00:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [421da39e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:00:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [421da39e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 5:00 PM >2013-03-10 17:00:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [421da39e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 5:00 PM >2013-03-10 17:00:02,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [68c3b638] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 185f6647 >2013-03-10 17:00:02,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [68c3b638] FINISH, GetHardwareInfoVDSCommand, log id: 185f6647 >2013-03-10 17:00:02,992 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-86) [2db7f99e] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 17:00:02,995 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-86) [71d776d6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 17:00:03,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1e48c63c >2013-03-10 17:01:09,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 17:01:09,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:01:09,048 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 17:01:09,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 1e48c63c >2013-03-10 17:01:09,052 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [73ee512] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:01:09,065 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [73ee512] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 74689d3d >2013-03-10 17:01:09,065 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [73ee512] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 17:01:09,065 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [73ee512] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 74689d3d >2013-03-10 17:01:09,070 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [71d776d6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 17:01:09,080 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-86) [2ec7ed96] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:01:09,082 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [2ec7ed96] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6d9262e5 >2013-03-10 17:01:09,083 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [2ec7ed96] FINISH, SetVdsStatusVDSCommand, log id: 6d9262e5 >2013-03-10 17:01:09,090 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-86) [2ec7ed96] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 17:01:09,111 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-86) [4446fdea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:01:09,114 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-86) [66ca51fd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:01:09,115 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-86) [66ca51fd] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 17:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [16503809] Autorecovering 1 hosts >2013-03-10 17:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [16503809] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [421326a6] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [421326a6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [421326a6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: ef2811 >2013-03-10 17:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [421326a6] FINISH, SetVdsStatusVDSCommand, log id: ef2811 >2013-03-10 17:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [421326a6] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2d0b721d >2013-03-10 17:05:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [421326a6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4e24ee63 >2013-03-10 17:05:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [421326a6] FINISH, GetHardwareInfoVDSCommand, log id: 4e24ee63 >2013-03-10 17:05:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [37642b6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:05:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [6db50e4a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:05:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [6db50e4a] FINISH, ActivateVdsVDSCommand, log id: 2d0b721d >2013-03-10 17:05:00,219 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [6db50e4a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:05:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-52) [6db50e4a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 17:05:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [6db50e4a] Autorecovering 1 storage domains >2013-03-10 17:05:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [6db50e4a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 17:05:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [51e5dc20] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:05:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [51e5dc20] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 5:05 PM >2013-03-10 17:05:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [51e5dc20] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 5:05 PM >2013-03-10 17:05:00,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [6687b8b8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4b84c4d9 >2013-03-10 17:05:00,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [6687b8b8] FINISH, GetHardwareInfoVDSCommand, log id: 4b84c4d9 >2013-03-10 17:05:00,356 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-24) [6f6e8a98] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 17:05:00,360 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-24) [1b383039] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 17:05:00,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 58bf6523 >2013-03-10 17:06:06,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 17:06:06,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:06:06,377 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 17:06:06,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 58bf6523 >2013-03-10 17:06:06,380 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [5c2c50d6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:06:06,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [5c2c50d6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1a77e8f6 >2013-03-10 17:06:06,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [5c2c50d6] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 17:06:06,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [5c2c50d6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1a77e8f6 >2013-03-10 17:06:06,396 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [5c2c50d6] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 17:06:06,397 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [1b383039] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 17:06:06,406 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-24) [1751a9a4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:06:06,407 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [1751a9a4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6dd07c9e >2013-03-10 17:06:06,409 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [1751a9a4] FINISH, SetVdsStatusVDSCommand, log id: 6dd07c9e >2013-03-10 17:06:06,414 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-24) [1751a9a4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 17:06:06,444 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-24) [4f021624] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:06:06,446 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-24) [2d03a309] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:06:06,448 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-24) [2d03a309] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 17:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [46af7501] Autorecovering 1 hosts >2013-03-10 17:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [46af7501] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [d00ffbc] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [d00ffbc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-29) [d00ffbc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3c924fbb >2013-03-10 17:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-29) [d00ffbc] FINISH, SetVdsStatusVDSCommand, log id: 3c924fbb >2013-03-10 17:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-29) [d00ffbc] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2589fc15 >2013-03-10 17:10:00,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-29) [d00ffbc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1f4fb9c3 >2013-03-10 17:10:00,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-29) [d00ffbc] FINISH, GetHardwareInfoVDSCommand, log id: 1f4fb9c3 >2013-03-10 17:10:00,457 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-29) [64f5df72] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:10:00,460 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-29) [4cf706f5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:10:00,461 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-29) [4cf706f5] FINISH, ActivateVdsVDSCommand, log id: 2589fc15 >2013-03-10 17:10:00,481 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [4cf706f5] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:10:00,482 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-29) [4cf706f5] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 17:10:00,483 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [4cf706f5] Autorecovering 1 storage domains >2013-03-10 17:10:00,483 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [4cf706f5] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 17:10:00,484 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [54cde22a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:10:00,484 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [54cde22a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 5:10 PM >2013-03-10 17:10:00,487 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [54cde22a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 5:10 PM >2013-03-10 17:10:00,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [293391a1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5e426bed >2013-03-10 17:10:00,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [293391a1] FINISH, GetHardwareInfoVDSCommand, log id: 5e426bed >2013-03-10 17:10:00,724 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-54) [789d99f8] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 17:10:00,727 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-54) [5d74cbb4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 17:10:00,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4a34b886 >2013-03-10 17:11:08,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 17:11:08,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:11:08,732 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 17:11:08,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 4a34b886 >2013-03-10 17:11:08,735 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [76168f49] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:11:08,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [76168f49] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 9b48cd7 >2013-03-10 17:11:08,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [76168f49] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 17:11:08,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [76168f49] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 9b48cd7 >2013-03-10 17:11:08,751 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [76168f49] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 17:11:08,752 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [5d74cbb4] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 17:11:08,761 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-54) [da451b4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:11:08,763 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [da451b4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1495093e >2013-03-10 17:11:08,764 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [da451b4] FINISH, SetVdsStatusVDSCommand, log id: 1495093e >2013-03-10 17:11:08,770 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [da451b4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 17:11:08,791 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [5f5c07e8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:11:08,794 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [7784180e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:11:08,795 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-54) [7784180e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 17:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-40) [90e1ee3] Autorecovering 1 hosts >2013-03-10 17:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-40) [90e1ee3] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-40) [4269d242] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-40) [4269d242] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-40) [4269d242] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 681f8e24 >2013-03-10 17:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-40) [4269d242] FINISH, SetVdsStatusVDSCommand, log id: 681f8e24 >2013-03-10 17:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-40) [4269d242] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1e459be4 >2013-03-10 17:15:00,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-40) [4269d242] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: b38de73 >2013-03-10 17:15:00,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-40) [4269d242] FINISH, GetHardwareInfoVDSCommand, log id: b38de73 >2013-03-10 17:15:00,184 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-40) [52111199] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:15:00,186 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-40) [3b81426c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:15:00,188 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-40) [3b81426c] FINISH, ActivateVdsVDSCommand, log id: 1e459be4 >2013-03-10 17:15:00,207 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-40) [3b81426c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:15:00,208 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-40) [3b81426c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 17:15:00,209 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-40) [3b81426c] Autorecovering 1 storage domains >2013-03-10 17:15:00,209 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-40) [3b81426c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 17:15:00,210 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-40) [39f1e06d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:15:00,210 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-40) [39f1e06d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 5:15 PM >2013-03-10 17:15:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-40) [39f1e06d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 5:15 PM >2013-03-10 17:15:02,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [5be3bae] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2d50cb66 >2013-03-10 17:15:03,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [5be3bae] FINISH, GetHardwareInfoVDSCommand, log id: 2d50cb66 >2013-03-10 17:15:03,041 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-87) [7cca988f] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 17:15:03,045 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-87) [5e44f5b8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 17:15:03,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7daa0725 >2013-03-10 17:16:07,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 17:16:07,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:16:07,026 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 17:16:07,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 7daa0725 >2013-03-10 17:16:07,030 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [a2f3508] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:16:07,044 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [a2f3508] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 11795c7c >2013-03-10 17:16:07,044 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [a2f3508] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 17:16:07,044 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [a2f3508] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 11795c7c >2013-03-10 17:16:07,046 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [a2f3508] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 17:16:07,047 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [5e44f5b8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 17:16:07,056 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-87) [7814d4b6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:16:07,057 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [7814d4b6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 72f10c3a >2013-03-10 17:16:07,059 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [7814d4b6] FINISH, SetVdsStatusVDSCommand, log id: 72f10c3a >2013-03-10 17:16:07,065 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [7814d4b6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 17:16:07,087 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [5fdea50] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:16:07,089 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [4bffb015] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:16:07,090 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-87) [4bffb015] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 17:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [2bda9e33] Autorecovering 1 hosts >2013-03-10 17:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [2bda9e33] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [1cb20fd9] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [1cb20fd9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [1cb20fd9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: d55b4b5 >2013-03-10 17:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [1cb20fd9] FINISH, SetVdsStatusVDSCommand, log id: d55b4b5 >2013-03-10 17:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-70) [1cb20fd9] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 37993ecf >2013-03-10 17:20:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [1cb20fd9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5980f1e6 >2013-03-10 17:20:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [1cb20fd9] FINISH, GetHardwareInfoVDSCommand, log id: 5980f1e6 >2013-03-10 17:20:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-70) [2c6b8348] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:20:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-70) [7b355f0e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:20:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-70) [7b355f0e] FINISH, ActivateVdsVDSCommand, log id: 37993ecf >2013-03-10 17:20:00,219 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [7b355f0e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:20:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-70) [7b355f0e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 17:20:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [7b355f0e] Autorecovering 1 storage domains >2013-03-10 17:20:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [7b355f0e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 17:20:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [41ad3119] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:20:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [41ad3119] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 5:20 PM >2013-03-10 17:20:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [41ad3119] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 5:20 PM >2013-03-10 17:20:01,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [1f3593ef] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4438759d >2013-03-10 17:20:01,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [1f3593ef] FINISH, GetHardwareInfoVDSCommand, log id: 4438759d >2013-03-10 17:20:01,333 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-5) [287f0eba] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 17:20:01,337 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-5) [48413a08] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 17:20:01,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 23ebc84a >2013-03-10 17:21:08,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 17:21:08,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:21:08,312 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 17:21:08,312 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 23ebc84a >2013-03-10 17:21:08,316 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [7ab22cd8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:21:08,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [7ab22cd8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1243f442 >2013-03-10 17:21:08,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [7ab22cd8] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 17:21:08,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [7ab22cd8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1243f442 >2013-03-10 17:21:08,331 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [7ab22cd8] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 17:21:08,331 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [48413a08] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 17:21:08,341 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-5) [12d9c6e1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:21:08,343 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [12d9c6e1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6fcb6e3 >2013-03-10 17:21:08,344 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [12d9c6e1] FINISH, SetVdsStatusVDSCommand, log id: 6fcb6e3 >2013-03-10 17:21:08,349 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-5) [12d9c6e1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 17:21:08,372 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-5) [2bc6db97] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:21:08,375 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-5) [516f9e35] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:21:08,376 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-5) [516f9e35] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 17:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [1aec2c2e] Autorecovering 1 hosts >2013-03-10 17:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [1aec2c2e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-12) [1a6c4a6e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-12) [1a6c4a6e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [1a6c4a6e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 32ff207b >2013-03-10 17:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [1a6c4a6e] FINISH, SetVdsStatusVDSCommand, log id: 32ff207b >2013-03-10 17:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-12) [1a6c4a6e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 433ebcbd >2013-03-10 17:25:00,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [1a6c4a6e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7554e42b >2013-03-10 17:25:00,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [1a6c4a6e] FINISH, GetHardwareInfoVDSCommand, log id: 7554e42b >2013-03-10 17:25:00,227 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-12) [569f6c5f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:25:00,229 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-12) [1974cacf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:25:00,231 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-12) [1974cacf] FINISH, ActivateVdsVDSCommand, log id: 433ebcbd >2013-03-10 17:25:00,250 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-12) [1974cacf] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:25:00,251 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-12) [1974cacf] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 17:25:00,252 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [1974cacf] Autorecovering 1 storage domains >2013-03-10 17:25:00,252 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [1974cacf] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 17:25:00,253 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-12) [5bddf247] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:25:00,253 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-12) [5bddf247] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 5:25 PM >2013-03-10 17:25:00,256 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-12) [5bddf247] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 5:25 PM >2013-03-10 17:25:02,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [6cfef8f0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4a5a0169 >2013-03-10 17:25:02,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [6cfef8f0] FINISH, GetHardwareInfoVDSCommand, log id: 4a5a0169 >2013-03-10 17:25:02,631 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-74) [383cf33e] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 17:25:02,635 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-74) [1f8052b9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 17:25:02,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: be9f19c >2013-03-10 17:26:09,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 17:26:09,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:26:09,636 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 17:26:09,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: be9f19c >2013-03-10 17:26:09,640 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [7860c8dc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:26:09,653 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [7860c8dc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 11b720d2 >2013-03-10 17:26:09,653 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [7860c8dc] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 17:26:09,653 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [7860c8dc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 11b720d2 >2013-03-10 17:26:09,655 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [7860c8dc] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 17:26:09,656 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [1f8052b9] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 17:26:09,666 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-74) [9637a5d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:26:09,667 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [9637a5d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 57b2ff8e >2013-03-10 17:26:09,668 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [9637a5d] FINISH, SetVdsStatusVDSCommand, log id: 57b2ff8e >2013-03-10 17:26:09,674 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-74) [9637a5d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 17:26:09,697 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-74) [178d14bf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:26:09,700 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-74) [7e0264f3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:26:09,701 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-74) [7e0264f3] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 17:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [3f7ba047] Autorecovering 1 hosts >2013-03-10 17:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [3f7ba047] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [68e390bf] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [68e390bf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [68e390bf] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5c20c6cf >2013-03-10 17:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [68e390bf] FINISH, SetVdsStatusVDSCommand, log id: 5c20c6cf >2013-03-10 17:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [68e390bf] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 295d78c1 >2013-03-10 17:30:00,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [68e390bf] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 216e6d95 >2013-03-10 17:30:00,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [68e390bf] FINISH, GetHardwareInfoVDSCommand, log id: 216e6d95 >2013-03-10 17:30:00,189 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-45) [484ce3b8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:30:00,192 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-45) [3ae3d998] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:30:00,193 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [3ae3d998] FINISH, ActivateVdsVDSCommand, log id: 295d78c1 >2013-03-10 17:30:00,212 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [3ae3d998] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:30:00,213 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-45) [3ae3d998] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 17:30:00,214 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [3ae3d998] Autorecovering 1 storage domains >2013-03-10 17:30:00,215 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [3ae3d998] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 17:30:00,215 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [2992ae79] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:30:00,215 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [2992ae79] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 5:30 PM >2013-03-10 17:30:00,218 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [2992ae79] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 5:30 PM >2013-03-10 17:30:00,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [797578a6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3f4adbae >2013-03-10 17:30:00,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [797578a6] FINISH, GetHardwareInfoVDSCommand, log id: 3f4adbae >2013-03-10 17:30:00,942 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-93) [441ce9c4] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 17:30:00,946 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-93) [297fcda7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 17:30:00,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 48ebdbf1 >2013-03-10 17:30:33,845 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-81) [5cb70b17] Start refreshing all users data >2013-03-10 17:30:34,995 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [1d2fdfa2] Attempting to update VMs/Templates Ovf. >2013-03-10 17:30:34,996 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [1d2fdfa2] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 17:30:34,997 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [1d2fdfa2] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 17:30:34,997 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [1d2fdfa2] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 17:30:34,998 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [1d2fdfa2] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 17:30:34,998 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [1d2fdfa2] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 17:30:34,999 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-76) [1d2fdfa2] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 17:31:08,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 17:31:08,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:31:08,998 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 17:31:08,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 48ebdbf1 >2013-03-10 17:31:09,002 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [31966f4a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:31:09,017 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [31966f4a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 22339f43 >2013-03-10 17:31:09,017 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [31966f4a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 17:31:09,017 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [31966f4a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 22339f43 >2013-03-10 17:31:09,019 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [31966f4a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 17:31:09,020 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [297fcda7] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 17:31:09,029 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-93) [374d6b64] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:31:09,031 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [374d6b64] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 37427fd2 >2013-03-10 17:31:09,032 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [374d6b64] FINISH, SetVdsStatusVDSCommand, log id: 37427fd2 >2013-03-10 17:31:09,038 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-93) [374d6b64] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 17:31:09,059 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [366b725] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:31:09,062 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [5c43f619] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:31:09,063 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-93) [5c43f619] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 17:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [2698864d] Autorecovering 1 hosts >2013-03-10 17:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [2698864d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [776518c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [776518c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [776518c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2186d61 >2013-03-10 17:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [776518c] FINISH, SetVdsStatusVDSCommand, log id: 2186d61 >2013-03-10 17:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-9) [776518c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4f0f522e >2013-03-10 17:35:00,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [776518c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4f211a67 >2013-03-10 17:35:00,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [776518c] FINISH, GetHardwareInfoVDSCommand, log id: 4f211a67 >2013-03-10 17:35:00,233 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-9) [397d57a8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:35:00,236 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-9) [27942f0c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:35:00,237 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-9) [27942f0c] FINISH, ActivateVdsVDSCommand, log id: 4f0f522e >2013-03-10 17:35:00,257 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [27942f0c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:35:00,258 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-9) [27942f0c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 17:35:00,259 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [27942f0c] Autorecovering 1 storage domains >2013-03-10 17:35:00,259 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [27942f0c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 17:35:00,260 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [2eccb85f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:35:00,260 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [2eccb85f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 5:35 PM >2013-03-10 17:35:00,263 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [2eccb85f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 5:35 PM >2013-03-10 17:35:00,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [23074657] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: b0b379f >2013-03-10 17:35:00,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [23074657] FINISH, GetHardwareInfoVDSCommand, log id: b0b379f >2013-03-10 17:35:00,335 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-63) [6b489ab3] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 17:35:00,338 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-63) [283aa728] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 17:35:00,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3233eff9 >2013-03-10 17:36:08,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 17:36:08,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:36:08,650 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 17:36:08,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 3233eff9 >2013-03-10 17:36:08,654 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [9ef097] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:36:08,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [9ef097] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 666004b8 >2013-03-10 17:36:08,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [9ef097] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 17:36:08,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [9ef097] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 666004b8 >2013-03-10 17:36:08,670 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [9ef097] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 17:36:08,671 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [283aa728] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 17:36:08,680 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-63) [2b373b7e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:36:08,681 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [2b373b7e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1bf327fa >2013-03-10 17:36:08,683 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [2b373b7e] FINISH, SetVdsStatusVDSCommand, log id: 1bf327fa >2013-03-10 17:36:08,689 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-63) [2b373b7e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 17:36:08,725 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-63) [72d4ecb4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:36:08,727 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-63) [7cdfe06c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:36:08,729 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-63) [7cdfe06c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 17:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [34a132dd] Autorecovering 1 hosts >2013-03-10 17:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [34a132dd] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [12e090c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [12e090c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [12e090c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7d725f75 >2013-03-10 17:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [12e090c] FINISH, SetVdsStatusVDSCommand, log id: 7d725f75 >2013-03-10 17:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-44) [12e090c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2f5e99c4 >2013-03-10 17:40:00,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [12e090c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7cd6e889 >2013-03-10 17:40:00,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [12e090c] FINISH, GetHardwareInfoVDSCommand, log id: 7cd6e889 >2013-03-10 17:40:00,453 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-44) [ca9c9c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:40:00,456 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-44) [7d2e4613] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:40:00,457 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-44) [7d2e4613] FINISH, ActivateVdsVDSCommand, log id: 2f5e99c4 >2013-03-10 17:40:00,476 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [7d2e4613] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:40:00,477 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-44) [7d2e4613] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 17:40:00,479 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [7d2e4613] Autorecovering 1 storage domains >2013-03-10 17:40:00,479 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [7d2e4613] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 17:40:00,479 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [2ee33b29] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:40:00,479 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [2ee33b29] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 5:40 PM >2013-03-10 17:40:00,482 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [2ee33b29] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 5:40 PM >2013-03-10 17:40:02,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [29f9f71f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 329361fe >2013-03-10 17:40:02,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [29f9f71f] FINISH, GetHardwareInfoVDSCommand, log id: 329361fe >2013-03-10 17:40:02,975 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-56) [4e2f9f7a] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 17:40:02,978 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-56) [116a7150] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 17:40:02,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6f306f28 >2013-03-10 17:41:08,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 17:41:08,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:41:08,989 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 17:41:08,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 6f306f28 >2013-03-10 17:41:08,993 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [682cc859] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:41:09,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [682cc859] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7038a4c5 >2013-03-10 17:41:09,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [682cc859] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 17:41:09,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [682cc859] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7038a4c5 >2013-03-10 17:41:09,010 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [682cc859] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 17:41:09,010 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [116a7150] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 17:41:09,020 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-56) [74e9711] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:41:09,021 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [74e9711] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 797deef4 >2013-03-10 17:41:09,023 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [74e9711] FINISH, SetVdsStatusVDSCommand, log id: 797deef4 >2013-03-10 17:41:09,029 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-56) [74e9711] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 17:41:09,051 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-56) [4a9af757] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:41:09,054 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-56) [794e0400] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:41:09,055 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-56) [794e0400] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 17:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [cc5193e] Autorecovering 1 hosts >2013-03-10 17:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [cc5193e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [1a088cf1] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [1a088cf1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [1a088cf1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 433ea9eb >2013-03-10 17:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [1a088cf1] FINISH, SetVdsStatusVDSCommand, log id: 433ea9eb >2013-03-10 17:45:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-22) [1a088cf1] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 55c29957 >2013-03-10 17:45:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [1a088cf1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 246b9d91 >2013-03-10 17:45:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [1a088cf1] FINISH, GetHardwareInfoVDSCommand, log id: 246b9d91 >2013-03-10 17:45:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [448515f2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:45:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [25b59857] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:45:00,203 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-22) [25b59857] FINISH, ActivateVdsVDSCommand, log id: 55c29957 >2013-03-10 17:45:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [25b59857] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:45:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [25b59857] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 17:45:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [25b59857] Autorecovering 1 storage domains >2013-03-10 17:45:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [25b59857] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 17:45:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [21b2cbb7] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:45:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [21b2cbb7] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 5:45 PM >2013-03-10 17:45:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [21b2cbb7] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 5:45 PM >2013-03-10 17:45:00,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [1a8af0b6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 60602187 >2013-03-10 17:45:00,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [1a8af0b6] FINISH, GetHardwareInfoVDSCommand, log id: 60602187 >2013-03-10 17:45:00,303 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-61) [7888ec5b] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 17:45:00,307 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-61) [2cbe13df] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 17:45:00,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 200c966b >2013-03-10 17:46:06,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 17:46:06,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:46:06,284 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 17:46:06,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 200c966b >2013-03-10 17:46:06,287 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [5e0b309] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:46:06,306 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [5e0b309] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7de9f474 >2013-03-10 17:46:06,306 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [5e0b309] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 17:46:06,306 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [5e0b309] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7de9f474 >2013-03-10 17:46:06,308 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [5e0b309] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 17:46:06,309 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [2cbe13df] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 17:46:06,320 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-61) [efe0d33] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:46:06,322 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [efe0d33] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 603c490d >2013-03-10 17:46:06,323 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [efe0d33] FINISH, SetVdsStatusVDSCommand, log id: 603c490d >2013-03-10 17:46:06,329 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-61) [efe0d33] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 17:46:06,353 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-61) [3811700a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:46:06,356 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-61) [3f9d3d8f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:46:06,357 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-61) [3f9d3d8f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 17:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [326dbb02] Autorecovering 1 hosts >2013-03-10 17:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [326dbb02] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [4542d389] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [4542d389] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [4542d389] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4399c6fb >2013-03-10 17:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [4542d389] FINISH, SetVdsStatusVDSCommand, log id: 4399c6fb >2013-03-10 17:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-46) [4542d389] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7533fc3e >2013-03-10 17:50:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [4542d389] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 52a3dbc8 >2013-03-10 17:50:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [4542d389] FINISH, GetHardwareInfoVDSCommand, log id: 52a3dbc8 >2013-03-10 17:50:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-46) [2d97ca6e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:50:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-46) [3dd35342] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:50:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-46) [3dd35342] FINISH, ActivateVdsVDSCommand, log id: 7533fc3e >2013-03-10 17:50:00,221 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [3dd35342] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:50:00,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-46) [3dd35342] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 17:50:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [3dd35342] Autorecovering 1 storage domains >2013-03-10 17:50:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [3dd35342] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 17:50:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [7702598c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:50:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [7702598c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 5:50 PM >2013-03-10 17:50:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [7702598c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 5:50 PM >2013-03-10 17:50:00,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [6e9b1af2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1b73ddf5 >2013-03-10 17:50:00,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [6e9b1af2] FINISH, GetHardwareInfoVDSCommand, log id: 1b73ddf5 >2013-03-10 17:50:00,599 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-57) [33fce4b] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 17:50:00,603 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-57) [58f39e30] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 17:50:00,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 451156d4 >2013-03-10 17:51:07,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 17:51:07,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:51:07,591 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 17:51:07,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 451156d4 >2013-03-10 17:51:07,595 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [7ab15323] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:51:07,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [7ab15323] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5a14d777 >2013-03-10 17:51:07,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [7ab15323] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 17:51:07,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [7ab15323] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5a14d777 >2013-03-10 17:51:07,612 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [7ab15323] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 17:51:07,613 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [58f39e30] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 17:51:07,622 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-57) [5514a811] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:51:07,623 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [5514a811] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6ab4ae35 >2013-03-10 17:51:07,625 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [5514a811] FINISH, SetVdsStatusVDSCommand, log id: 6ab4ae35 >2013-03-10 17:51:07,631 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-57) [5514a811] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 17:51:07,655 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-57) [78c5ff6a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:51:07,658 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-57) [ea4e15b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:51:07,659 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-57) [ea4e15b] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 17:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [76a558c3] Autorecovering 1 hosts >2013-03-10 17:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [76a558c3] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [58bb8baf] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [58bb8baf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [58bb8baf] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 795846c9 >2013-03-10 17:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [58bb8baf] FINISH, SetVdsStatusVDSCommand, log id: 795846c9 >2013-03-10 17:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-65) [58bb8baf] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 28ec9195 >2013-03-10 17:55:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [58bb8baf] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 224cb765 >2013-03-10 17:55:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [58bb8baf] FINISH, GetHardwareInfoVDSCommand, log id: 224cb765 >2013-03-10 17:55:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-65) [52a44d6b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:55:00,262 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-65) [63a8f507] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:55:00,264 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-65) [63a8f507] FINISH, ActivateVdsVDSCommand, log id: 28ec9195 >2013-03-10 17:55:00,283 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [63a8f507] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 17:55:00,284 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-65) [63a8f507] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 17:55:00,286 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [63a8f507] Autorecovering 1 storage domains >2013-03-10 17:55:00,286 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [63a8f507] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 17:55:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [f6aa27f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:55:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [f6aa27f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 5:55 PM >2013-03-10 17:55:00,290 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [f6aa27f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 5:55 PM >2013-03-10 17:55:01,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [763e1d0c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6d9ec6de >2013-03-10 17:55:01,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [763e1d0c] FINISH, GetHardwareInfoVDSCommand, log id: 6d9ec6de >2013-03-10 17:55:01,907 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-1) [78dedc58] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 17:55:01,911 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-1) [d787a02] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 17:55:01,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 58920af7 >2013-03-10 17:56:09,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 17:56:09,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 17:56:09,933 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 17:56:09,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 58920af7 >2013-03-10 17:56:09,937 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [3f176bd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 17:56:09,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [3f176bd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 41652153 >2013-03-10 17:56:09,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [3f176bd] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 17:56:09,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [3f176bd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 41652153 >2013-03-10 17:56:09,953 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [3f176bd] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 17:56:09,953 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [d787a02] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 17:56:09,963 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-1) [684fdaab] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:56:09,964 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [684fdaab] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 57c58d74 >2013-03-10 17:56:09,965 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [684fdaab] FINISH, SetVdsStatusVDSCommand, log id: 57c58d74 >2013-03-10 17:56:09,971 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-1) [684fdaab] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 17:56:10,000 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-1) [448eb4e8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:56:10,003 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-1) [668aa132] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 17:56:10,004 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-1) [668aa132] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 18:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [39683fa4] Autorecovering 1 hosts >2013-03-10 18:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [39683fa4] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [4a8115f3] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [4a8115f3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [4a8115f3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 43c47f38 >2013-03-10 18:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [4a8115f3] FINISH, SetVdsStatusVDSCommand, log id: 43c47f38 >2013-03-10 18:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-35) [4a8115f3] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2e91d117 >2013-03-10 18:00:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [4a8115f3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3a1539b >2013-03-10 18:00:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [4a8115f3] FINISH, GetHardwareInfoVDSCommand, log id: 3a1539b >2013-03-10 18:00:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-35) [76757072] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:00:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-35) [35c411b8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:00:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-35) [35c411b8] FINISH, ActivateVdsVDSCommand, log id: 2e91d117 >2013-03-10 18:00:00,221 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [35c411b8] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:00:00,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-35) [35c411b8] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 18:00:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [35c411b8] Autorecovering 1 storage domains >2013-03-10 18:00:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [35c411b8] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 18:00:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [19cfa4ed] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:00:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [19cfa4ed] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 6:00 PM >2013-03-10 18:00:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [19cfa4ed] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 6:00 PM >2013-03-10 18:00:01,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [7784180e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6c6e692b >2013-03-10 18:00:01,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [7784180e] FINISH, GetHardwareInfoVDSCommand, log id: 6c6e692b >2013-03-10 18:00:01,255 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-54) [4ddef239] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 18:00:01,258 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-54) [41e745ef] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 18:00:01,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: d95a862 >2013-03-10 18:01:08,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 18:01:08,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:01:08,306 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 18:01:08,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: d95a862 >2013-03-10 18:01:08,310 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [4d50cfe5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:01:08,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [4d50cfe5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2ae4da9d >2013-03-10 18:01:08,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [4d50cfe5] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 18:01:08,324 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [4d50cfe5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2ae4da9d >2013-03-10 18:01:08,326 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [4d50cfe5] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 18:01:08,326 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [41e745ef] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 18:01:08,336 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-54) [4de1ec05] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:01:08,338 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [4de1ec05] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 438e5d41 >2013-03-10 18:01:08,339 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [4de1ec05] FINISH, SetVdsStatusVDSCommand, log id: 438e5d41 >2013-03-10 18:01:08,345 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [4de1ec05] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 18:01:08,367 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [5f86d111] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:01:08,370 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [5855b52] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:01:08,371 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-54) [5855b52] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 18:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [4bffb015] Autorecovering 1 hosts >2013-03-10 18:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [4bffb015] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [1dc5dcb7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [1dc5dcb7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [1dc5dcb7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 47a5bc62 >2013-03-10 18:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [1dc5dcb7] FINISH, SetVdsStatusVDSCommand, log id: 47a5bc62 >2013-03-10 18:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-87) [1dc5dcb7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 20ddec2a >2013-03-10 18:05:00,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [1dc5dcb7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 580a3099 >2013-03-10 18:05:00,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [1dc5dcb7] FINISH, GetHardwareInfoVDSCommand, log id: 580a3099 >2013-03-10 18:05:00,247 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [58c8b9b2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:05:00,249 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [38222762] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:05:00,250 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-87) [38222762] FINISH, ActivateVdsVDSCommand, log id: 20ddec2a >2013-03-10 18:05:00,269 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [38222762] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:05:00,270 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [38222762] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 18:05:00,272 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [38222762] Autorecovering 1 storage domains >2013-03-10 18:05:00,272 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [38222762] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 18:05:00,272 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [1bed0da9] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:05:00,273 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [1bed0da9] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 6:05 PM >2013-03-10 18:05:00,275 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [1bed0da9] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 6:05 PM >2013-03-10 18:05:02,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [5329e9ba] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5a085324 >2013-03-10 18:05:02,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [5329e9ba] FINISH, GetHardwareInfoVDSCommand, log id: 5a085324 >2013-03-10 18:05:02,644 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-4) [483890e7] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 18:05:02,647 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-4) [42a2d3f6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 18:05:02,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7ba35401 >2013-03-10 18:06:09,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 18:06:09,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:06:09,672 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 18:06:09,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 7ba35401 >2013-03-10 18:06:09,676 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [35caafee] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:06:09,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [35caafee] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1cf4cc8 >2013-03-10 18:06:09,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [35caafee] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 18:06:09,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [35caafee] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1cf4cc8 >2013-03-10 18:06:09,695 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [42a2d3f6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 18:06:09,705 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-4) [bf9316f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:06:09,706 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [bf9316f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3284f611 >2013-03-10 18:06:09,707 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [bf9316f] FINISH, SetVdsStatusVDSCommand, log id: 3284f611 >2013-03-10 18:06:09,721 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-4) [bf9316f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 18:06:09,744 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-4) [2efa45f8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:06:09,746 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-4) [54ec0185] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:06:09,747 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-4) [54ec0185] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 18:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [ee9c6e6] Autorecovering 1 hosts >2013-03-10 18:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [ee9c6e6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [655f49b0] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [655f49b0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [655f49b0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 32218f83 >2013-03-10 18:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [655f49b0] FINISH, SetVdsStatusVDSCommand, log id: 32218f83 >2013-03-10 18:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-58) [655f49b0] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6c98e2d9 >2013-03-10 18:10:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [655f49b0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 27e673e >2013-03-10 18:10:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [655f49b0] FINISH, GetHardwareInfoVDSCommand, log id: 27e673e >2013-03-10 18:10:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-58) [4a7551ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:10:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-58) [68cb4fb1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:10:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-58) [68cb4fb1] FINISH, ActivateVdsVDSCommand, log id: 6c98e2d9 >2013-03-10 18:10:00,221 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [68cb4fb1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:10:00,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-58) [68cb4fb1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 18:10:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [68cb4fb1] Autorecovering 1 storage domains >2013-03-10 18:10:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [68cb4fb1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 18:10:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [3cfd3cf] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:10:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [3cfd3cf] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 6:10 PM >2013-03-10 18:10:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [3cfd3cf] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 6:10 PM >2013-03-10 18:10:00,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [5d712191] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7e08d2a >2013-03-10 18:10:00,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [5d712191] FINISH, GetHardwareInfoVDSCommand, log id: 7e08d2a >2013-03-10 18:10:00,991 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-38) [7f1da86f] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 18:10:00,994 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-38) [b578df0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 18:10:01,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: cd2d5b3 >2013-03-10 18:10:48,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 18:10:48,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:10:48,668 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 18:10:48,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: cd2d5b3 >2013-03-10 18:10:48,672 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [67197dac] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:10:48,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [67197dac] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1ec2bd32 >2013-03-10 18:10:48,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [67197dac] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 18:10:48,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [67197dac] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1ec2bd32 >2013-03-10 18:10:48,688 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [67197dac] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 18:10:48,689 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [b578df0] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 18:10:48,698 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-38) [5fd7712] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:10:48,699 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [5fd7712] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7f87e79d >2013-03-10 18:10:48,700 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [5fd7712] FINISH, SetVdsStatusVDSCommand, log id: 7f87e79d >2013-03-10 18:10:48,706 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-38) [5fd7712] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 18:10:48,728 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-38) [4c4c62ad] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:10:48,730 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-38) [655d7eb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:10:48,732 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-38) [655d7eb] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 18:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [421da39e] Autorecovering 1 hosts >2013-03-10 18:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [421da39e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [6862c1ad] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [6862c1ad] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [6862c1ad] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 58949c91 >2013-03-10 18:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [6862c1ad] FINISH, SetVdsStatusVDSCommand, log id: 58949c91 >2013-03-10 18:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-88) [6862c1ad] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: e3616f >2013-03-10 18:15:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [6862c1ad] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 46ab4605 >2013-03-10 18:15:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [6862c1ad] FINISH, GetHardwareInfoVDSCommand, log id: 46ab4605 >2013-03-10 18:15:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-88) [27dbf4fb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:15:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-88) [42bee855] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:15:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-88) [42bee855] FINISH, ActivateVdsVDSCommand, log id: e3616f >2013-03-10 18:15:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [42bee855] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:15:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-88) [42bee855] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 18:15:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [42bee855] Autorecovering 1 storage domains >2013-03-10 18:15:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [42bee855] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 18:15:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [24a55380] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:15:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [24a55380] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 6:15 PM >2013-03-10 18:15:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [24a55380] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 6:15 PM >2013-03-10 18:15:00,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [449a10e6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 18e86ac >2013-03-10 18:15:00,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [449a10e6] FINISH, GetHardwareInfoVDSCommand, log id: 18e86ac >2013-03-10 18:15:00,983 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-13) [13aa2f58] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 18:15:00,986 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-13) [277634c4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 18:15:00,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2ec4998e >2013-03-10 18:15:09,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 18:15:09,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:15:09,970 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 18:15:09,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 2ec4998e >2013-03-10 18:15:09,974 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [4f4db7f1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:15:09,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4f4db7f1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1f7c3d6f >2013-03-10 18:15:09,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [4f4db7f1] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 18:15:09,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4f4db7f1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1f7c3d6f >2013-03-10 18:15:09,990 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [4f4db7f1] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 18:15:09,990 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [277634c4] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 18:15:10,000 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-13) [3c386584] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:15:10,002 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [3c386584] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6be930c6 >2013-03-10 18:15:10,003 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [3c386584] FINISH, SetVdsStatusVDSCommand, log id: 6be930c6 >2013-03-10 18:15:10,010 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-13) [3c386584] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 18:15:10,039 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-13) [5c57c5cb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:15:10,042 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-13) [2d4f7649] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:15:10,043 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-13) [2d4f7649] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 18:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [38ad567d] Autorecovering 1 hosts >2013-03-10 18:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [38ad567d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [2e31f0ef] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [2e31f0ef] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [2e31f0ef] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 31ef066a >2013-03-10 18:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [2e31f0ef] FINISH, SetVdsStatusVDSCommand, log id: 31ef066a >2013-03-10 18:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [2e31f0ef] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2f00b922 >2013-03-10 18:20:00,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [2e31f0ef] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 333cc037 >2013-03-10 18:20:00,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [2e31f0ef] FINISH, GetHardwareInfoVDSCommand, log id: 333cc037 >2013-03-10 18:20:00,526 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-84) [3ec3807] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:20:00,529 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-84) [b3bfa48] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:20:00,530 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [b3bfa48] FINISH, ActivateVdsVDSCommand, log id: 2f00b922 >2013-03-10 18:20:00,549 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [b3bfa48] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:20:00,550 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-84) [b3bfa48] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 18:20:00,552 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [b3bfa48] Autorecovering 1 storage domains >2013-03-10 18:20:00,552 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [b3bfa48] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 18:20:00,552 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [43751bef] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:20:00,552 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [43751bef] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 6:20 PM >2013-03-10 18:20:00,555 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [43751bef] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 6:20 PM >2013-03-10 18:20:01,272 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [1a3ead3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 16aeb279 >2013-03-10 18:20:01,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [1a3ead3] FINISH, GetHardwareInfoVDSCommand, log id: 16aeb279 >2013-03-10 18:20:01,330 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-97) [e0a9fc4] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 18:20:01,334 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-97) [7c45417f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 18:20:01,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5aba404b >2013-03-10 18:20:10,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 18:20:10,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:20:10,335 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 18:20:10,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 5aba404b >2013-03-10 18:20:10,339 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [44e582bf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:20:10,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [44e582bf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 58a6ae03 >2013-03-10 18:20:10,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [44e582bf] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 18:20:10,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [44e582bf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 58a6ae03 >2013-03-10 18:20:10,355 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [44e582bf] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 18:20:10,355 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [7c45417f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 18:20:10,365 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-97) [35f94826] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:20:10,366 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [35f94826] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 421af04c >2013-03-10 18:20:10,367 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [35f94826] FINISH, SetVdsStatusVDSCommand, log id: 421af04c >2013-03-10 18:20:10,373 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-97) [35f94826] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 18:20:10,395 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-97) [7342f464] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:20:10,398 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-97) [77351aaa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:20:10,399 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-97) [77351aaa] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 18:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [7d6b044d] Autorecovering 1 hosts >2013-03-10 18:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [7d6b044d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [1ee3f894] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [1ee3f894] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [1ee3f894] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 545c2ae1 >2013-03-10 18:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [1ee3f894] FINISH, SetVdsStatusVDSCommand, log id: 545c2ae1 >2013-03-10 18:25:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [1ee3f894] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 18a2b9e4 >2013-03-10 18:25:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [1ee3f894] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7b92edb7 >2013-03-10 18:25:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [1ee3f894] FINISH, GetHardwareInfoVDSCommand, log id: 7b92edb7 >2013-03-10 18:25:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-42) [674e6fad] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:25:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-42) [6c00d2ab] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:25:00,093 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [6c00d2ab] FINISH, ActivateVdsVDSCommand, log id: 18a2b9e4 >2013-03-10 18:25:00,113 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [6c00d2ab] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:25:00,114 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-42) [6c00d2ab] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 18:25:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [6c00d2ab] Autorecovering 1 storage domains >2013-03-10 18:25:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [6c00d2ab] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 18:25:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [24ca6dc2] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:25:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [24ca6dc2] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 6:25 PM >2013-03-10 18:25:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [24ca6dc2] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 6:25 PM >2013-03-10 18:25:01,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [5bddf247] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7627fa31 >2013-03-10 18:25:01,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [5bddf247] FINISH, GetHardwareInfoVDSCommand, log id: 7627fa31 >2013-03-10 18:25:01,679 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-12) [1166e526] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 18:25:01,682 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-12) [54f4c12b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 18:25:01,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7f35976f >2013-03-10 18:25:05,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 18:25:05,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:25:05,671 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 18:25:05,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 7f35976f >2013-03-10 18:25:05,675 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [721bf1cc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:25:05,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [721bf1cc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 577f1ab9 >2013-03-10 18:25:05,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [721bf1cc] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 18:25:05,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [721bf1cc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 577f1ab9 >2013-03-10 18:25:05,691 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [721bf1cc] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 18:25:05,692 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [54f4c12b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 18:25:05,701 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-12) [127bf7a6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:25:05,702 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [127bf7a6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 49c02b33 >2013-03-10 18:25:05,704 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [127bf7a6] FINISH, SetVdsStatusVDSCommand, log id: 49c02b33 >2013-03-10 18:25:05,710 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-12) [127bf7a6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 18:25:05,733 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-12) [239624d2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:25:05,735 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-12) [2590f959] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:25:05,737 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-12) [2590f959] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 18:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [1348b24b] Autorecovering 1 hosts >2013-03-10 18:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [1348b24b] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [5b491af1] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [5b491af1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [5b491af1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 69ecb579 >2013-03-10 18:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [5b491af1] FINISH, SetVdsStatusVDSCommand, log id: 69ecb579 >2013-03-10 18:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [5b491af1] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 70a40d65 >2013-03-10 18:30:00,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [5b491af1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4d08ab0e >2013-03-10 18:30:00,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [5b491af1] FINISH, GetHardwareInfoVDSCommand, log id: 4d08ab0e >2013-03-10 18:30:00,101 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [4908e025] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:30:00,103 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [7ceeda57] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:30:00,104 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [7ceeda57] FINISH, ActivateVdsVDSCommand, log id: 70a40d65 >2013-03-10 18:30:00,131 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [7ceeda57] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:30:00,132 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [7ceeda57] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 18:30:00,134 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [7ceeda57] Autorecovering 1 storage domains >2013-03-10 18:30:00,134 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [7ceeda57] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 18:30:00,134 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [52260984] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:30:00,135 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [52260984] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 6:30 PM >2013-03-10 18:30:00,137 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [52260984] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 6:30 PM >2013-03-10 18:30:02,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [463f5475] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5815937b >2013-03-10 18:30:03,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [463f5475] FINISH, GetHardwareInfoVDSCommand, log id: 5815937b >2013-03-10 18:30:03,050 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-7) [75d96160] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 18:30:03,054 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-7) [5d9b6bac] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 18:30:03,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 38736410 >2013-03-10 18:30:08,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 18:30:08,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:30:08,088 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 18:30:08,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 38736410 >2013-03-10 18:30:08,092 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [358e226e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:30:08,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [358e226e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1b9b1d08 >2013-03-10 18:30:08,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [358e226e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 18:30:08,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [358e226e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1b9b1d08 >2013-03-10 18:30:08,108 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [358e226e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 18:30:08,108 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [5d9b6bac] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 18:30:08,117 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-7) [614938c2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:30:08,119 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [614938c2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 9785c20 >2013-03-10 18:30:08,120 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [614938c2] FINISH, SetVdsStatusVDSCommand, log id: 9785c20 >2013-03-10 18:30:08,126 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-7) [614938c2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 18:30:08,150 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-7) [2ce2291b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:30:08,152 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-7) [1cf38a99] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:30:08,154 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) [1cf38a99] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 18:30:33,847 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-12) [2590f959] Start refreshing all users data >2013-03-10 18:30:35,000 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [4d280997] Attempting to update VMs/Templates Ovf. >2013-03-10 18:30:35,001 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [4d280997] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 18:30:35,002 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [4d280997] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 18:30:35,002 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [4d280997] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 18:30:35,002 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [4d280997] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 18:30:35,002 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [4d280997] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 18:30:35,003 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [4d280997] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 18:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [43751bef] Autorecovering 1 hosts >2013-03-10 18:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [43751bef] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [53100216] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [53100216] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [53100216] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3ab76137 >2013-03-10 18:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [53100216] FINISH, SetVdsStatusVDSCommand, log id: 3ab76137 >2013-03-10 18:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [53100216] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: be3e731 >2013-03-10 18:35:02,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [53100216] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 571eb6d2 >2013-03-10 18:35:02,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [53100216] FINISH, GetHardwareInfoVDSCommand, log id: 571eb6d2 >2013-03-10 18:35:02,443 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-84) [66fe9eb8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:35:02,445 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-84) [3c9df461] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:35:02,447 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [3c9df461] FINISH, ActivateVdsVDSCommand, log id: be3e731 >2013-03-10 18:35:02,466 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [3c9df461] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:35:02,467 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-84) [3c9df461] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 18:35:02,468 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [3c9df461] Autorecovering 1 storage domains >2013-03-10 18:35:02,468 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [3c9df461] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 18:35:02,469 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [7ed08cc5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:35:02,469 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [7ed08cc5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 6:35 PM >2013-03-10 18:35:02,472 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [7ed08cc5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 6:35 PM >2013-03-10 18:35:02,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [56ef438c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 66449bfa >2013-03-10 18:35:02,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [56ef438c] FINISH, GetHardwareInfoVDSCommand, log id: 66449bfa >2013-03-10 18:35:02,545 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-92) [39cfed84] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 18:35:02,548 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-92) [4fbd1660] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 18:35:02,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2d3e7481 >2013-03-10 18:35:08,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 18:35:08,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:35:08,842 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 18:35:08,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 2d3e7481 >2013-03-10 18:35:08,845 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [8087298] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:35:08,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [8087298] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3d729bfd >2013-03-10 18:35:08,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [8087298] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 18:35:08,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [8087298] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3d729bfd >2013-03-10 18:35:08,862 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [8087298] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 18:35:08,863 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [4fbd1660] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 18:35:08,872 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-92) [6b721fe8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:35:08,874 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [6b721fe8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1c1867af >2013-03-10 18:35:08,875 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [6b721fe8] FINISH, SetVdsStatusVDSCommand, log id: 1c1867af >2013-03-10 18:35:08,881 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-92) [6b721fe8] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 18:35:08,950 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-92) [470e881f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:35:08,952 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-92) [17bcf0cf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:35:08,954 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-92) [17bcf0cf] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 18:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [24ca6dc2] Autorecovering 1 hosts >2013-03-10 18:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [24ca6dc2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [19f6e5a7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [19f6e5a7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [19f6e5a7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 9aa4be1 >2013-03-10 18:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [19f6e5a7] FINISH, SetVdsStatusVDSCommand, log id: 9aa4be1 >2013-03-10 18:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [19f6e5a7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6fd40508 >2013-03-10 18:40:00,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [19f6e5a7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1df99edd >2013-03-10 18:40:00,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [19f6e5a7] FINISH, GetHardwareInfoVDSCommand, log id: 1df99edd >2013-03-10 18:40:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-42) [d28a978] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:40:00,094 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-42) [6c029cb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:40:00,095 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [6c029cb] FINISH, ActivateVdsVDSCommand, log id: 6fd40508 >2013-03-10 18:40:00,114 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [6c029cb] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:40:00,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-42) [6c029cb] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 18:40:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [6c029cb] Autorecovering 1 storage domains >2013-03-10 18:40:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [6c029cb] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 18:40:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [48e21c0c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:40:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [48e21c0c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 6:40 PM >2013-03-10 18:40:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [48e21c0c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 6:40 PM >2013-03-10 18:40:00,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [a2d4d2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2c1cf9f0 >2013-03-10 18:40:00,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [a2d4d2] FINISH, GetHardwareInfoVDSCommand, log id: 2c1cf9f0 >2013-03-10 18:40:00,234 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-37) [5de67b37] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 18:40:00,238 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-37) [19b83bab] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 18:40:00,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 53b82e1a >2013-03-10 18:40:07,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 18:40:07,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:40:07,245 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 18:40:07,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 53b82e1a >2013-03-10 18:40:07,249 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [740cb269] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:40:07,263 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [740cb269] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2e736f23 >2013-03-10 18:40:07,263 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [740cb269] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 18:40:07,263 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [740cb269] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2e736f23 >2013-03-10 18:40:07,265 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [740cb269] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 18:40:07,265 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [19b83bab] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 18:40:07,275 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-37) [6f863cd0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:40:07,277 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [6f863cd0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: f8725f5 >2013-03-10 18:40:07,278 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [6f863cd0] FINISH, SetVdsStatusVDSCommand, log id: f8725f5 >2013-03-10 18:40:07,285 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-37) [6f863cd0] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 18:40:07,307 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-37) [24c04af4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:40:07,310 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-37) [764ac8be] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:40:07,311 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-37) [764ac8be] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 18:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [4c090c7d] Autorecovering 1 hosts >2013-03-10 18:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [4c090c7d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [5cb95cd1] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [5cb95cd1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [5cb95cd1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 514f0046 >2013-03-10 18:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [5cb95cd1] FINISH, SetVdsStatusVDSCommand, log id: 514f0046 >2013-03-10 18:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-78) [5cb95cd1] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3b17d332 >2013-03-10 18:45:00,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [5cb95cd1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 78f3ccee >2013-03-10 18:45:00,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [5cb95cd1] FINISH, GetHardwareInfoVDSCommand, log id: 78f3ccee >2013-03-10 18:45:00,404 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-78) [377d3863] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:45:00,407 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-78) [78836a7b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:45:00,408 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-78) [78836a7b] FINISH, ActivateVdsVDSCommand, log id: 3b17d332 >2013-03-10 18:45:00,427 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [78836a7b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:45:00,428 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-78) [78836a7b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 18:45:00,429 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [78836a7b] Autorecovering 1 storage domains >2013-03-10 18:45:00,430 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [78836a7b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 18:45:00,430 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [553c3ff2] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:45:00,430 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [553c3ff2] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 6:45 PM >2013-03-10 18:45:00,433 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [553c3ff2] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 6:45 PM >2013-03-10 18:45:01,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [24a55380] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1a9eb7a4 >2013-03-10 18:45:01,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [24a55380] FINISH, GetHardwareInfoVDSCommand, log id: 1a9eb7a4 >2013-03-10 18:45:01,597 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-88) [648d456a] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 18:45:01,601 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-88) [52c36918] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 18:45:01,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 42d1fe04 >2013-03-10 18:45:08,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 18:45:08,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:45:08,587 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 18:45:08,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 42d1fe04 >2013-03-10 18:45:08,591 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [3334b20a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:45:08,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [3334b20a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 673a17cb >2013-03-10 18:45:08,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [3334b20a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 18:45:08,605 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [3334b20a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 673a17cb >2013-03-10 18:45:08,607 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [3334b20a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 18:45:08,607 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [52c36918] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 18:45:08,618 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-88) [1761e60] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:45:08,619 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [1761e60] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 761b85ee >2013-03-10 18:45:08,621 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [1761e60] FINISH, SetVdsStatusVDSCommand, log id: 761b85ee >2013-03-10 18:45:08,628 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-88) [1761e60] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 18:45:08,658 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-88) [62ed74f5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:45:08,661 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-88) [6f93f931] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:45:08,662 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-88) [6f93f931] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 18:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [1510a5b2] Autorecovering 1 hosts >2013-03-10 18:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [1510a5b2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-91) [56cdbaa2] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-91) [56cdbaa2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:50:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [56cdbaa2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6b0ab9f1 >2013-03-10 18:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [56cdbaa2] FINISH, SetVdsStatusVDSCommand, log id: 6b0ab9f1 >2013-03-10 18:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-91) [56cdbaa2] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2f9c9393 >2013-03-10 18:50:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [56cdbaa2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 491a45e1 >2013-03-10 18:50:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [56cdbaa2] FINISH, GetHardwareInfoVDSCommand, log id: 491a45e1 >2013-03-10 18:50:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-91) [686e7393] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:50:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-91) [6e7ee142] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:50:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-91) [6e7ee142] FINISH, ActivateVdsVDSCommand, log id: 2f9c9393 >2013-03-10 18:50:00,221 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-91) [6e7ee142] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:50:00,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-91) [6e7ee142] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 18:50:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [6e7ee142] Autorecovering 1 storage domains >2013-03-10 18:50:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [6e7ee142] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 18:50:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-91) [67c686fb] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:50:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-91) [67c686fb] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 6:50 PM >2013-03-10 18:50:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-91) [67c686fb] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 6:50 PM >2013-03-10 18:50:02,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [56bc550] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 59d14b6d >2013-03-10 18:50:02,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [56bc550] FINISH, GetHardwareInfoVDSCommand, log id: 59d14b6d >2013-03-10 18:50:02,950 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-47) [72b39062] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 18:50:02,953 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-47) [77abfe47] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 18:50:02,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 19c9641a >2013-03-10 18:50:10,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 18:50:10,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:50:10,948 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 18:50:10,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 19c9641a >2013-03-10 18:50:10,952 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [d3f0a95] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:50:10,965 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [d3f0a95] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 22b5eebd >2013-03-10 18:50:10,965 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [d3f0a95] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 18:50:10,965 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [d3f0a95] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 22b5eebd >2013-03-10 18:50:10,967 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [d3f0a95] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 18:50:10,968 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [77abfe47] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 18:50:10,977 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-47) [6d7bb2bf] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:50:10,979 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [6d7bb2bf] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 38b9621d >2013-03-10 18:50:10,980 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [6d7bb2bf] FINISH, SetVdsStatusVDSCommand, log id: 38b9621d >2013-03-10 18:50:10,987 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [6d7bb2bf] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 18:50:11,014 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [3b0afb17] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:50:11,017 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [71aefce4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:50:11,018 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-47) [71aefce4] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 18:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [ea4e15b] Autorecovering 1 hosts >2013-03-10 18:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [ea4e15b] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [757f1369] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [757f1369] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [757f1369] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7204f155 >2013-03-10 18:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [757f1369] FINISH, SetVdsStatusVDSCommand, log id: 7204f155 >2013-03-10 18:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-57) [757f1369] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6bae950d >2013-03-10 18:55:00,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [757f1369] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: b476239 >2013-03-10 18:55:00,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [757f1369] FINISH, GetHardwareInfoVDSCommand, log id: b476239 >2013-03-10 18:55:00,402 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-57) [4ce288ce] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:55:00,405 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-57) [1de80ac1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:55:00,406 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-57) [1de80ac1] FINISH, ActivateVdsVDSCommand, log id: 6bae950d >2013-03-10 18:55:00,425 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [1de80ac1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 18:55:00,426 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-57) [1de80ac1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 18:55:00,428 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [1de80ac1] Autorecovering 1 storage domains >2013-03-10 18:55:00,428 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [1de80ac1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 18:55:00,428 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [41f2564c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:55:00,428 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [41f2564c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 6:55 PM >2013-03-10 18:55:00,431 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [41f2564c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 6:55 PM >2013-03-10 18:55:02,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [48e21c0c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6ce705c1 >2013-03-10 18:55:02,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [48e21c0c] FINISH, GetHardwareInfoVDSCommand, log id: 6ce705c1 >2013-03-10 18:55:02,304 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-42) [33e5a3d1] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 18:55:02,308 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-42) [50b332dc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 18:55:02,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 61f82837 >2013-03-10 18:55:09,329 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 18:55:09,329 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 18:55:09,329 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 18:55:09,329 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 61f82837 >2013-03-10 18:55:09,333 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [1136b098] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 18:55:09,347 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [1136b098] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 471d303 >2013-03-10 18:55:09,347 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [1136b098] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 18:55:09,347 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [1136b098] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 471d303 >2013-03-10 18:55:09,349 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [1136b098] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 18:55:09,349 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [50b332dc] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 18:55:09,359 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-42) [74a8df7f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:55:09,361 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [74a8df7f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 52a88832 >2013-03-10 18:55:09,362 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [74a8df7f] FINISH, SetVdsStatusVDSCommand, log id: 52a88832 >2013-03-10 18:55:09,367 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-42) [74a8df7f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 18:55:09,390 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-42) [65af3f92] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:55:09,393 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-42) [2e09c15a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 18:55:09,394 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-42) [2e09c15a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 19:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [7cdfe06c] Autorecovering 1 hosts >2013-03-10 19:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [7cdfe06c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:00:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [41e05431] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:00:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [41e05431] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:00:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [41e05431] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2d626873 >2013-03-10 19:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [41e05431] FINISH, SetVdsStatusVDSCommand, log id: 2d626873 >2013-03-10 19:00:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [41e05431] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: fdcf2c6 >2013-03-10 19:00:00,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [41e05431] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2c853f7 >2013-03-10 19:00:00,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [41e05431] FINISH, GetHardwareInfoVDSCommand, log id: 2c853f7 >2013-03-10 19:00:00,417 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-63) [7bc30e4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:00:00,419 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-63) [6a5f7390] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:00:00,421 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [6a5f7390] FINISH, ActivateVdsVDSCommand, log id: fdcf2c6 >2013-03-10 19:00:00,440 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [6a5f7390] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:00:00,441 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-63) [6a5f7390] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 19:00:00,442 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [6a5f7390] Autorecovering 1 storage domains >2013-03-10 19:00:00,442 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [6a5f7390] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 19:00:00,443 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [57b553a4] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:00:00,443 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [57b553a4] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 7:00 PM >2013-03-10 19:00:00,446 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [57b553a4] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 7:00 PM >2013-03-10 19:00:00,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [76b925b4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 53ac92fa >2013-03-10 19:00:00,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [76b925b4] FINISH, GetHardwareInfoVDSCommand, log id: 53ac92fa >2013-03-10 19:00:00,668 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-18) [5300454d] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 19:00:00,672 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-18) [2ed76a04] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 19:00:00,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6368391a >2013-03-10 19:00:07,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 19:00:07,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:00:07,710 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 19:00:07,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 6368391a >2013-03-10 19:00:07,714 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [733203f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:00:07,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [733203f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 294ee9df >2013-03-10 19:00:07,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [733203f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 19:00:07,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [733203f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 294ee9df >2013-03-10 19:00:07,730 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [733203f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 19:00:07,730 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [2ed76a04] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 19:00:07,740 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-18) [23964e7d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:00:07,742 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [23964e7d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1ea8fa74 >2013-03-10 19:00:07,743 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [23964e7d] FINISH, SetVdsStatusVDSCommand, log id: 1ea8fa74 >2013-03-10 19:00:07,750 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [23964e7d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 19:00:07,771 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [42a647bd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:00:07,773 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [422e650f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:00:07,775 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) [422e650f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 19:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [689413c1] Autorecovering 1 hosts >2013-03-10 19:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [689413c1] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-48) [6db990c7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-48) [6db990c7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [6db990c7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 39b60bc2 >2013-03-10 19:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [6db990c7] FINISH, SetVdsStatusVDSCommand, log id: 39b60bc2 >2013-03-10 19:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-48) [6db990c7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6cefc170 >2013-03-10 19:05:00,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [6db990c7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 14c8f9ed >2013-03-10 19:05:00,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [6db990c7] FINISH, GetHardwareInfoVDSCommand, log id: 14c8f9ed >2013-03-10 19:05:00,186 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-48) [3d90a5e7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:05:00,189 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-48) [165a0f28] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:05:00,190 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-48) [165a0f28] FINISH, ActivateVdsVDSCommand, log id: 6cefc170 >2013-03-10 19:05:00,209 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-48) [165a0f28] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:05:00,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-48) [165a0f28] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 19:05:00,211 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [165a0f28] Autorecovering 1 storage domains >2013-03-10 19:05:00,211 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [165a0f28] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 19:05:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-48) [3c8afe7d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:05:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-48) [3c8afe7d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 7:05 PM >2013-03-10 19:05:00,215 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-48) [3c8afe7d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 7:05 PM >2013-03-10 19:05:02,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [67c686fb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 520a6fa1 >2013-03-10 19:05:02,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [67c686fb] FINISH, GetHardwareInfoVDSCommand, log id: 520a6fa1 >2013-03-10 19:05:02,063 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-91) [1a99679b] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 19:05:02,066 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-91) [68b2b42e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 19:05:02,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3071e841 >2013-03-10 19:05:10,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 19:05:10,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:05:10,080 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 19:05:10,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 3071e841 >2013-03-10 19:05:10,083 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [1f445be4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:05:10,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [1f445be4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1384ad70 >2013-03-10 19:05:10,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [1f445be4] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 19:05:10,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [1f445be4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1384ad70 >2013-03-10 19:05:10,100 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [1f445be4] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 19:05:10,100 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [68b2b42e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 19:05:10,109 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-91) [2815e7c4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:05:10,110 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [2815e7c4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 60ec4d08 >2013-03-10 19:05:10,112 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [2815e7c4] FINISH, SetVdsStatusVDSCommand, log id: 60ec4d08 >2013-03-10 19:05:10,118 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-91) [2815e7c4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 19:05:10,140 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-91) [35ebe823] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:05:10,143 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-91) [4ce3d552] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:05:10,144 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-91) [4ce3d552] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 19:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [3f6684f9] Autorecovering 1 hosts >2013-03-10 19:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [3f6684f9] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [2b6c48fd] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [2b6c48fd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [2b6c48fd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7732d5fe >2013-03-10 19:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [2b6c48fd] FINISH, SetVdsStatusVDSCommand, log id: 7732d5fe >2013-03-10 19:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [2b6c48fd] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 109ca25a >2013-03-10 19:10:00,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [2b6c48fd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7fdc4143 >2013-03-10 19:10:00,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [2b6c48fd] FINISH, GetHardwareInfoVDSCommand, log id: 7fdc4143 >2013-03-10 19:10:00,306 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [44012226] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:10:00,308 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [91a4199] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:10:00,310 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [91a4199] FINISH, ActivateVdsVDSCommand, log id: 109ca25a >2013-03-10 19:10:00,329 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [91a4199] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:10:00,330 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-94) [91a4199] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 19:10:00,331 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [91a4199] Autorecovering 1 storage domains >2013-03-10 19:10:00,331 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [91a4199] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 19:10:00,332 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [351681a3] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:10:00,332 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [351681a3] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 7:10 PM >2013-03-10 19:10:00,335 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [351681a3] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 7:10 PM >2013-03-10 19:10:01,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [39c82f66] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5bf07cd5 >2013-03-10 19:10:01,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [39c82f66] FINISH, GetHardwareInfoVDSCommand, log id: 5bf07cd5 >2013-03-10 19:10:01,431 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-66) [2866f229] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 19:10:01,434 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-66) [6a37dc73] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 19:10:01,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 59d7c1de >2013-03-10 19:10:08,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 19:10:08,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:10:08,444 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 19:10:08,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 59d7c1de >2013-03-10 19:10:08,448 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [6eb1d1a2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:10:08,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [6eb1d1a2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2545dc23 >2013-03-10 19:10:08,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [6eb1d1a2] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 19:10:08,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [6eb1d1a2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2545dc23 >2013-03-10 19:10:08,467 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [6a37dc73] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 19:10:08,476 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-66) [5c9a0318] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:10:08,478 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [5c9a0318] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6d71392a >2013-03-10 19:10:08,479 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [5c9a0318] FINISH, SetVdsStatusVDSCommand, log id: 6d71392a >2013-03-10 19:10:08,485 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-66) [5c9a0318] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 19:10:08,506 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-66) [5932ef1b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:10:08,508 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-66) [675dc8bd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:10:08,509 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-66) [675dc8bd] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 19:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [4520582c] Autorecovering 1 hosts >2013-03-10 19:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [4520582c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [4de8bd11] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [4de8bd11] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [4de8bd11] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 18b12c2f >2013-03-10 19:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [4de8bd11] FINISH, SetVdsStatusVDSCommand, log id: 18b12c2f >2013-03-10 19:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-64) [4de8bd11] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2558ad94 >2013-03-10 19:15:00,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [4de8bd11] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1829df5b >2013-03-10 19:15:00,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [4de8bd11] FINISH, GetHardwareInfoVDSCommand, log id: 1829df5b >2013-03-10 19:15:00,539 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-64) [2779aba2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:15:00,541 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-64) [24161ae] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:15:00,543 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-64) [24161ae] FINISH, ActivateVdsVDSCommand, log id: 2558ad94 >2013-03-10 19:15:00,562 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [24161ae] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:15:00,563 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-64) [24161ae] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 19:15:00,564 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [24161ae] Autorecovering 1 storage domains >2013-03-10 19:15:00,564 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [24161ae] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 19:15:00,565 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [929b8aa] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:15:00,565 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [929b8aa] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 7:15 PM >2013-03-10 19:15:00,568 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [929b8aa] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 7:15 PM >2013-03-10 19:15:02,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [52260984] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 76b48c02 >2013-03-10 19:15:02,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [52260984] FINISH, GetHardwareInfoVDSCommand, log id: 76b48c02 >2013-03-10 19:15:02,784 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-59) [1ea9681e] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 19:15:02,788 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-59) [61bc85e9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 19:15:02,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3a184bb8 >2013-03-10 19:15:10,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 19:15:10,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:15:10,413 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 19:15:10,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 3a184bb8 >2013-03-10 19:15:10,417 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [4a64b6a5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:15:10,430 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4a64b6a5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7111c116 >2013-03-10 19:15:10,430 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [4a64b6a5] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 19:15:10,430 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4a64b6a5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7111c116 >2013-03-10 19:15:10,432 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [4a64b6a5] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 19:15:10,433 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [61bc85e9] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 19:15:10,442 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-59) [48ead96f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:15:10,443 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [48ead96f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2709f49b >2013-03-10 19:15:10,445 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [48ead96f] FINISH, SetVdsStatusVDSCommand, log id: 2709f49b >2013-03-10 19:15:10,450 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [48ead96f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 19:15:10,471 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [44c880f8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:15:10,474 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [7a58471c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:15:10,475 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-59) [7a58471c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 19:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [19cfa4ed] Autorecovering 1 hosts >2013-03-10 19:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [19cfa4ed] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [7d61541a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [7d61541a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [7d61541a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5faa496d >2013-03-10 19:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [7d61541a] FINISH, SetVdsStatusVDSCommand, log id: 5faa496d >2013-03-10 19:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-35) [7d61541a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 23f64ab4 >2013-03-10 19:20:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [7d61541a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 39b9d33b >2013-03-10 19:20:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [7d61541a] FINISH, GetHardwareInfoVDSCommand, log id: 39b9d33b >2013-03-10 19:20:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-35) [20b0a856] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:20:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-35) [65580b02] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:20:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-35) [65580b02] FINISH, ActivateVdsVDSCommand, log id: 23f64ab4 >2013-03-10 19:20:00,112 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [65580b02] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:20:00,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-35) [65580b02] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 19:20:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [65580b02] Autorecovering 1 storage domains >2013-03-10 19:20:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [65580b02] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 19:20:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [69fe74ad] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:20:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [69fe74ad] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 7:20 PM >2013-03-10 19:20:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [69fe74ad] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 7:20 PM >2013-03-10 19:20:01,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [4fe38e0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: a29bd7e >2013-03-10 19:20:01,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [4fe38e0] FINISH, GetHardwareInfoVDSCommand, log id: a29bd7e >2013-03-10 19:20:01,753 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-55) [1c456ac9] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 19:20:01,757 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-55) [aac3e0f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 19:20:01,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7d4a3e91 >2013-03-10 19:20:10,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 19:20:10,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:20:10,773 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 19:20:10,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 7d4a3e91 >2013-03-10 19:20:10,776 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [7283ff4e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:20:10,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [7283ff4e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7f56db8d >2013-03-10 19:20:10,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [7283ff4e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 19:20:10,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [7283ff4e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7f56db8d >2013-03-10 19:20:10,792 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [7283ff4e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 19:20:10,793 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [aac3e0f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 19:20:10,802 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-55) [749ece20] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:20:10,804 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [749ece20] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 34f7281 >2013-03-10 19:20:10,805 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [749ece20] FINISH, SetVdsStatusVDSCommand, log id: 34f7281 >2013-03-10 19:20:10,810 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-55) [749ece20] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 19:20:10,832 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-55) [66ca5c39] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:20:10,835 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-55) [77078b90] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:20:10,837 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-55) [77078b90] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 19:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [c6191f9] Autorecovering 1 hosts >2013-03-10 19:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [c6191f9] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-53) [4a31ebdb] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-53) [4a31ebdb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:25:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [4a31ebdb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 655e6c2f >2013-03-10 19:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [4a31ebdb] FINISH, SetVdsStatusVDSCommand, log id: 655e6c2f >2013-03-10 19:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-53) [4a31ebdb] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: ac30baa >2013-03-10 19:25:00,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [4a31ebdb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 532e7fa4 >2013-03-10 19:25:00,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [4a31ebdb] FINISH, GetHardwareInfoVDSCommand, log id: 532e7fa4 >2013-03-10 19:25:00,087 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-53) [42de8abd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:25:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-53) [f4bc755] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:25:00,091 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-53) [f4bc755] FINISH, ActivateVdsVDSCommand, log id: ac30baa >2013-03-10 19:25:00,110 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-53) [f4bc755] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:25:00,111 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-53) [f4bc755] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 19:25:00,112 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [f4bc755] Autorecovering 1 storage domains >2013-03-10 19:25:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [f4bc755] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 19:25:00,113 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-53) [7b2426a9] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:25:00,113 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-53) [7b2426a9] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 7:25 PM >2013-03-10 19:25:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-53) [7b2426a9] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 7:25 PM >2013-03-10 19:25:02,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [5855b52] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3c932710 >2013-03-10 19:25:02,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [5855b52] FINISH, GetHardwareInfoVDSCommand, log id: 3c932710 >2013-03-10 19:25:02,117 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-54) [2edfc092] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 19:25:02,121 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-54) [5d20513c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 19:25:02,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3e708582 >2013-03-10 19:25:09,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 19:25:09,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:25:09,116 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 19:25:09,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 3e708582 >2013-03-10 19:25:09,120 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [6b70181b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:25:09,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [6b70181b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 22c349c0 >2013-03-10 19:25:09,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [6b70181b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 19:25:09,134 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [6b70181b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 22c349c0 >2013-03-10 19:25:09,136 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [6b70181b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 19:25:09,137 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [5d20513c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 19:25:09,146 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-54) [7a2ba8a1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:25:09,148 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [7a2ba8a1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7949ace3 >2013-03-10 19:25:09,149 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [7a2ba8a1] FINISH, SetVdsStatusVDSCommand, log id: 7949ace3 >2013-03-10 19:25:09,155 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [7a2ba8a1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 19:25:09,177 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [61f0d3c2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:25:09,180 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [73847da4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:25:09,181 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-54) [73847da4] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 19:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) [3f9d3d8f] Autorecovering 1 hosts >2013-03-10 19:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) [3f9d3d8f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-61) [1716e71b] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-61) [1716e71b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [1716e71b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 21027c85 >2013-03-10 19:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [1716e71b] FINISH, SetVdsStatusVDSCommand, log id: 21027c85 >2013-03-10 19:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-61) [1716e71b] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 476e554d >2013-03-10 19:30:00,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [1716e71b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 707e271a >2013-03-10 19:30:00,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [1716e71b] FINISH, GetHardwareInfoVDSCommand, log id: 707e271a >2013-03-10 19:30:00,094 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-61) [30b55703] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:30:00,097 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-61) [830c9d7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:30:00,098 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-61) [830c9d7] FINISH, ActivateVdsVDSCommand, log id: 476e554d >2013-03-10 19:30:00,117 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-61) [830c9d7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:30:00,118 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-61) [830c9d7] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 19:30:00,119 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) [830c9d7] Autorecovering 1 storage domains >2013-03-10 19:30:00,119 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) [830c9d7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 19:30:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-61) [53ecd05b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:30:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-61) [53ecd05b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 7:30 PM >2013-03-10 19:30:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-61) [53ecd05b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 7:30 PM >2013-03-10 19:30:00,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [3755b11] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7d87d4b8 >2013-03-10 19:30:00,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [3755b11] FINISH, GetHardwareInfoVDSCommand, log id: 7d87d4b8 >2013-03-10 19:30:00,453 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-16) [22ac4d91] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 19:30:00,456 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-16) [1c3b87e2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 19:30:00,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 41737711 >2013-03-10 19:30:09,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 19:30:09,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:30:09,521 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 19:30:09,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 41737711 >2013-03-10 19:30:09,525 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [47e00ef6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:30:09,541 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [47e00ef6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 751f3d33 >2013-03-10 19:30:09,541 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [47e00ef6] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 19:30:09,541 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [47e00ef6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 751f3d33 >2013-03-10 19:30:09,543 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [47e00ef6] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 19:30:09,543 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [1c3b87e2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 19:30:09,553 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-16) [4ab10eca] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:30:09,554 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [4ab10eca] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 54297bda >2013-03-10 19:30:09,556 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [4ab10eca] FINISH, SetVdsStatusVDSCommand, log id: 54297bda >2013-03-10 19:30:09,562 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-16) [4ab10eca] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 19:30:09,585 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-16) [437889a7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:30:09,587 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-16) [2f79528e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:30:09,589 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-16) [2f79528e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 19:30:33,849 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-89) [7e74c320] Start refreshing all users data >2013-03-10 19:30:35,003 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Attempting to update VMs/Templates Ovf. >2013-03-10 19:30:35,004 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 19:30:35,005 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 19:30:35,005 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 19:30:35,006 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 19:30:35,006 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 19:30:35,006 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [52a39daf] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 19:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [2bb4e603] Autorecovering 1 hosts >2013-03-10 19:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [2bb4e603] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [1d737668] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [1d737668] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [1d737668] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 32828408 >2013-03-10 19:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [1d737668] FINISH, SetVdsStatusVDSCommand, log id: 32828408 >2013-03-10 19:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-83) [1d737668] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 79b9a22b >2013-03-10 19:35:00,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [1d737668] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2b21f37f >2013-03-10 19:35:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [1d737668] FINISH, GetHardwareInfoVDSCommand, log id: 2b21f37f >2013-03-10 19:35:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-83) [75425056] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:35:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-83) [59a176bf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:35:00,091 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-83) [59a176bf] FINISH, ActivateVdsVDSCommand, log id: 79b9a22b >2013-03-10 19:35:00,112 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [59a176bf] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:35:00,113 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-83) [59a176bf] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 19:35:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [59a176bf] Autorecovering 1 storage domains >2013-03-10 19:35:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [59a176bf] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 19:35:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [192b01be] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:35:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [192b01be] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 7:35 PM >2013-03-10 19:35:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [192b01be] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 7:35 PM >2013-03-10 19:35:00,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [18811a0a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5c0df775 >2013-03-10 19:35:00,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [18811a0a] FINISH, GetHardwareInfoVDSCommand, log id: 5c0df775 >2013-03-10 19:35:00,894 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-69) [189b3b7d] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 19:35:00,898 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-69) [5b263949] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 19:35:00,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 198d6cd >2013-03-10 19:35:08,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 19:35:08,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:35:08,890 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 19:35:08,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 198d6cd >2013-03-10 19:35:08,893 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [63172a07] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:35:08,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [63172a07] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5f9c923e >2013-03-10 19:35:08,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [63172a07] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 19:35:08,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [63172a07] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5f9c923e >2013-03-10 19:35:08,910 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [63172a07] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 19:35:08,911 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [5b263949] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 19:35:08,921 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-69) [30231fcb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:35:08,922 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [30231fcb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1cd13072 >2013-03-10 19:35:08,924 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [30231fcb] FINISH, SetVdsStatusVDSCommand, log id: 1cd13072 >2013-03-10 19:35:08,929 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-69) [30231fcb] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 19:35:08,951 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-69) [4c50651f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:35:08,953 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-69) [2f45233e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:35:08,954 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-69) [2f45233e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 19:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [40c4fe46] Autorecovering 1 hosts >2013-03-10 19:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [40c4fe46] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [6c02c4cf] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [6c02c4cf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [6c02c4cf] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 10668dd7 >2013-03-10 19:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [6c02c4cf] FINISH, SetVdsStatusVDSCommand, log id: 10668dd7 >2013-03-10 19:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-41) [6c02c4cf] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 77c58133 >2013-03-10 19:40:00,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [6c02c4cf] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5e4aa17a >2013-03-10 19:40:00,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [6c02c4cf] FINISH, GetHardwareInfoVDSCommand, log id: 5e4aa17a >2013-03-10 19:40:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-41) [74f91693] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:40:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-41) [3f9bc308] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:40:00,093 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-41) [3f9bc308] FINISH, ActivateVdsVDSCommand, log id: 77c58133 >2013-03-10 19:40:00,113 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [3f9bc308] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:40:00,114 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-41) [3f9bc308] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 19:40:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [3f9bc308] Autorecovering 1 storage domains >2013-03-10 19:40:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [3f9bc308] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 19:40:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [353d0c29] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:40:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [353d0c29] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 7:40 PM >2013-03-10 19:40:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [353d0c29] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 7:40 PM >2013-03-10 19:40:00,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [602f56c8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 75bc58b9 >2013-03-10 19:40:00,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [602f56c8] FINISH, GetHardwareInfoVDSCommand, log id: 75bc58b9 >2013-03-10 19:40:00,234 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-32) [3cb86921] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 19:40:00,237 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-32) [3588cddb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 19:40:00,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 17e2435d >2013-03-10 19:40:07,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 19:40:07,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:40:07,238 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 19:40:07,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 17e2435d >2013-03-10 19:40:07,242 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [1f85b5b6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:40:07,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [1f85b5b6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2bc80ac1 >2013-03-10 19:40:07,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [1f85b5b6] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 19:40:07,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [1f85b5b6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2bc80ac1 >2013-03-10 19:40:07,258 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [1f85b5b6] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 19:40:07,258 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [3588cddb] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 19:40:07,268 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-32) [55885971] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:40:07,269 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [55885971] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3ffda26 >2013-03-10 19:40:07,271 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [55885971] FINISH, SetVdsStatusVDSCommand, log id: 3ffda26 >2013-03-10 19:40:07,276 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-32) [55885971] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 19:40:07,298 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-32) [b256121] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:40:07,300 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-32) [f9076ac] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:40:07,302 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-32) [f9076ac] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 19:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [8897c9b] Autorecovering 1 hosts >2013-03-10 19:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [8897c9b] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [720d7ea4] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [720d7ea4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [720d7ea4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 14808b00 >2013-03-10 19:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [720d7ea4] FINISH, SetVdsStatusVDSCommand, log id: 14808b00 >2013-03-10 19:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [720d7ea4] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 275fb47b >2013-03-10 19:45:00,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [720d7ea4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 282e1c8d >2013-03-10 19:45:00,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [720d7ea4] FINISH, GetHardwareInfoVDSCommand, log id: 282e1c8d >2013-03-10 19:45:00,302 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [38c31b97] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:45:00,305 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [6406394f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:45:00,306 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [6406394f] FINISH, ActivateVdsVDSCommand, log id: 275fb47b >2013-03-10 19:45:00,325 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [6406394f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:45:00,326 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-8) [6406394f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 19:45:00,327 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [6406394f] Autorecovering 1 storage domains >2013-03-10 19:45:00,327 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [6406394f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 19:45:00,328 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [44b03170] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:45:00,328 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [44b03170] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 7:45 PM >2013-03-10 19:45:00,331 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [44b03170] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 7:45 PM >2013-03-10 19:45:01,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-62) [7685c42e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 693e9157 >2013-03-10 19:45:01,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-62) [7685c42e] FINISH, GetHardwareInfoVDSCommand, log id: 693e9157 >2013-03-10 19:45:01,587 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-62) [21cf9a4] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 19:45:01,590 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-62) [44cf2697] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 19:45:01,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: b42fd8e >2013-03-10 19:45:09,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 19:45:09,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:45:09,573 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 19:45:09,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: b42fd8e >2013-03-10 19:45:09,577 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [4f495f73] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:45:09,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [4f495f73] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 8bf1968 >2013-03-10 19:45:09,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [4f495f73] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 19:45:09,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [4f495f73] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 8bf1968 >2013-03-10 19:45:09,593 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [4f495f73] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 19:45:09,593 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [44cf2697] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 19:45:09,603 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-62) [105f70fa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:45:09,604 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-62) [105f70fa] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 79a1df70 >2013-03-10 19:45:09,605 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-62) [105f70fa] FINISH, SetVdsStatusVDSCommand, log id: 79a1df70 >2013-03-10 19:45:09,611 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-62) [105f70fa] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 19:45:09,633 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-62) [75f71494] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:45:09,635 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-62) [41f2f67a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:45:09,636 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-62) [41f2f67a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 19:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [3cfd3cf] Autorecovering 1 hosts >2013-03-10 19:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [3cfd3cf] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [334c7b2a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [334c7b2a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [334c7b2a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3c56c981 >2013-03-10 19:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [334c7b2a] FINISH, SetVdsStatusVDSCommand, log id: 3c56c981 >2013-03-10 19:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-58) [334c7b2a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 209fc44e >2013-03-10 19:50:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [334c7b2a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7face728 >2013-03-10 19:50:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [334c7b2a] FINISH, GetHardwareInfoVDSCommand, log id: 7face728 >2013-03-10 19:50:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-58) [5b50469b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:50:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-58) [5dec4cf2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:50:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-58) [5dec4cf2] FINISH, ActivateVdsVDSCommand, log id: 209fc44e >2013-03-10 19:50:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [5dec4cf2] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:50:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-58) [5dec4cf2] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 19:50:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [5dec4cf2] Autorecovering 1 storage domains >2013-03-10 19:50:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [5dec4cf2] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 19:50:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [4760c562] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:50:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [4760c562] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 7:50 PM >2013-03-10 19:50:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [4760c562] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 7:50 PM >2013-03-10 19:50:00,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [516f9e35] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1dd0d307 >2013-03-10 19:50:00,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [516f9e35] FINISH, GetHardwareInfoVDSCommand, log id: 1dd0d307 >2013-03-10 19:50:00,915 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-5) [48916207] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 19:50:00,919 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-5) [66aa0d44] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 19:50:00,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 52575983 >2013-03-10 19:50:06,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 19:50:06,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:50:06,919 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 19:50:06,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 52575983 >2013-03-10 19:50:06,923 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [3c82e493] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:50:06,936 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [3c82e493] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2aef2946 >2013-03-10 19:50:06,936 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [3c82e493] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 19:50:06,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [3c82e493] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2aef2946 >2013-03-10 19:50:06,939 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [3c82e493] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 19:50:06,939 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [66aa0d44] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 19:50:06,949 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-5) [3ffe8197] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:50:06,950 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [3ffe8197] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2c07759e >2013-03-10 19:50:06,952 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [3ffe8197] FINISH, SetVdsStatusVDSCommand, log id: 2c07759e >2013-03-10 19:50:06,958 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-5) [3ffe8197] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 19:50:06,980 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-5) [6f62c34] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:50:06,982 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-5) [571972d3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:50:06,984 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-5) [571972d3] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 19:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [f6aa27f] Autorecovering 1 hosts >2013-03-10 19:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [f6aa27f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [52a1fd74] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [52a1fd74] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [52a1fd74] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5366d6c >2013-03-10 19:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [52a1fd74] FINISH, SetVdsStatusVDSCommand, log id: 5366d6c >2013-03-10 19:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-65) [52a1fd74] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 9d965a6 >2013-03-10 19:55:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [52a1fd74] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3ab64275 >2013-03-10 19:55:00,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [52a1fd74] FINISH, GetHardwareInfoVDSCommand, log id: 3ab64275 >2013-03-10 19:55:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-65) [6f9be99d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:55:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-65) [584611ba] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:55:00,205 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-65) [584611ba] FINISH, ActivateVdsVDSCommand, log id: 9d965a6 >2013-03-10 19:55:00,224 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [584611ba] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 19:55:00,225 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-65) [584611ba] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 19:55:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [584611ba] Autorecovering 1 storage domains >2013-03-10 19:55:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [584611ba] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 19:55:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [4ef5030d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:55:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [4ef5030d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 7:55 PM >2013-03-10 19:55:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [4ef5030d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 7:55 PM >2013-03-10 19:55:01,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [3803c270] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7f44465e >2013-03-10 19:55:01,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [3803c270] FINISH, GetHardwareInfoVDSCommand, log id: 7f44465e >2013-03-10 19:55:01,266 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-73) [3a268c1a] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 19:55:01,270 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-73) [5fe7517e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 19:55:01,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 54cd1e3b >2013-03-10 19:55:08,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 19:55:08,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 19:55:08,264 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 19:55:08,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 54cd1e3b >2013-03-10 19:55:08,268 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [7103baec] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 19:55:08,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [7103baec] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1806b5f6 >2013-03-10 19:55:08,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [7103baec] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 19:55:08,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [7103baec] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1806b5f6 >2013-03-10 19:55:08,284 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [7103baec] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 19:55:08,285 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [5fe7517e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 19:55:08,294 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-73) [3fac5998] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:55:08,296 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [3fac5998] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 79a14ed >2013-03-10 19:55:08,297 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [3fac5998] FINISH, SetVdsStatusVDSCommand, log id: 79a14ed >2013-03-10 19:55:08,303 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-73) [3fac5998] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 19:55:08,325 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-73) [3eebf470] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:55:08,327 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-73) [53f93e14] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 19:55:08,329 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-73) [53f93e14] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 20:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [3f8e56c4] Autorecovering 1 hosts >2013-03-10 20:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [3f8e56c4] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [33da1258] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [33da1258] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [33da1258] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 436b008f >2013-03-10 20:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [33da1258] FINISH, SetVdsStatusVDSCommand, log id: 436b008f >2013-03-10 20:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-27) [33da1258] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3304f970 >2013-03-10 20:00:00,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [33da1258] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 79c44c28 >2013-03-10 20:00:00,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [33da1258] FINISH, GetHardwareInfoVDSCommand, log id: 79c44c28 >2013-03-10 20:00:00,415 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-27) [447e5c22] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:00:00,418 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-27) [7658cf9b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:00:00,419 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-27) [7658cf9b] FINISH, ActivateVdsVDSCommand, log id: 3304f970 >2013-03-10 20:00:00,438 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [7658cf9b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:00:00,442 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [7658cf9b] Autorecovering 1 storage domains >2013-03-10 20:00:00,442 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [7658cf9b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 20:00:00,443 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [7cac8d58] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:00:00,443 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [7cac8d58] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 8:00 PM >2013-03-10 20:00:00,446 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [7cac8d58] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 8:00 PM >2013-03-10 20:00:02,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [5663ec8f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1c7d99b6 >2013-03-10 20:00:02,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [5663ec8f] FINISH, GetHardwareInfoVDSCommand, log id: 1c7d99b6 >2013-03-10 20:00:02,605 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-26) [4ef9ad79] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 20:00:02,609 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-26) [6feb3001] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 20:00:02,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 41371125 >2013-03-10 20:00:07,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 20:00:07,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:00:07,645 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 20:00:07,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 41371125 >2013-03-10 20:00:07,648 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [c449dca] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:00:07,662 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [c449dca] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3c735bfe >2013-03-10 20:00:07,662 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [c449dca] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 20:00:07,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [c449dca] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3c735bfe >2013-03-10 20:00:07,665 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [c449dca] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 20:00:07,665 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [6feb3001] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 20:00:07,675 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-26) [512d6275] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:00:07,677 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [512d6275] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 115a3eae >2013-03-10 20:00:07,678 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [512d6275] FINISH, SetVdsStatusVDSCommand, log id: 115a3eae >2013-03-10 20:00:07,683 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [512d6275] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 20:00:07,704 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [25333403] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:00:07,707 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [547b8320] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:00:07,708 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-26) [547b8320] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 20:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [266664c7] Autorecovering 1 hosts >2013-03-10 20:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [266664c7] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [5995af8b] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [5995af8b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [5995af8b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1c4ccd49 >2013-03-10 20:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [5995af8b] FINISH, SetVdsStatusVDSCommand, log id: 1c4ccd49 >2013-03-10 20:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-19) [5995af8b] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 66da30ec >2013-03-10 20:05:00,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [5995af8b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4b37bf00 >2013-03-10 20:05:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [5995af8b] FINISH, GetHardwareInfoVDSCommand, log id: 4b37bf00 >2013-03-10 20:05:00,257 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-19) [3c90f4c7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:05:00,259 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-19) [293a9ea1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:05:00,261 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-19) [293a9ea1] FINISH, ActivateVdsVDSCommand, log id: 66da30ec >2013-03-10 20:05:00,280 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [293a9ea1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:05:00,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-19) [293a9ea1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 20:05:00,282 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [293a9ea1] Autorecovering 1 storage domains >2013-03-10 20:05:00,282 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [293a9ea1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 20:05:00,283 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [7a5aeb22] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:05:00,283 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [7a5aeb22] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 8:05 PM >2013-03-10 20:05:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [7a5aeb22] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 8:05 PM >2013-03-10 20:05:01,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [51e5dc20] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4139796b >2013-03-10 20:05:01,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [51e5dc20] FINISH, GetHardwareInfoVDSCommand, log id: 4139796b >2013-03-10 20:05:01,989 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-52) [64dc3b85] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 20:05:01,992 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-52) [546c6928] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 20:05:02,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 37fc1fc >2013-03-10 20:05:11,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 20:05:11,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:05:11,005 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 20:05:11,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 37fc1fc >2013-03-10 20:05:11,009 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [7bc7c11b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:05:11,022 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [7bc7c11b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4cccb223 >2013-03-10 20:05:11,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [7bc7c11b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 20:05:11,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [7bc7c11b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4cccb223 >2013-03-10 20:05:11,025 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [7bc7c11b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 20:05:11,025 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [546c6928] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 20:05:11,034 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-52) [3ec032f3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:05:11,036 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [3ec032f3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5af1f804 >2013-03-10 20:05:11,037 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [3ec032f3] FINISH, SetVdsStatusVDSCommand, log id: 5af1f804 >2013-03-10 20:05:11,043 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-52) [3ec032f3] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 20:05:11,065 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [38ac2ce6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:05:11,068 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [7998d983] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:05:11,069 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-52) [7998d983] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 20:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [2d4f7649] Autorecovering 1 hosts >2013-03-10 20:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [2d4f7649] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [345ecfe9] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [345ecfe9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [345ecfe9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1d9ac659 >2013-03-10 20:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [345ecfe9] FINISH, SetVdsStatusVDSCommand, log id: 1d9ac659 >2013-03-10 20:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-13) [345ecfe9] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 267d9c2c >2013-03-10 20:10:00,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [345ecfe9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3de1f3b0 >2013-03-10 20:10:00,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [345ecfe9] FINISH, GetHardwareInfoVDSCommand, log id: 3de1f3b0 >2013-03-10 20:10:00,233 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-13) [77d6082d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:10:00,236 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-13) [4aa1184c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:10:00,237 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-13) [4aa1184c] FINISH, ActivateVdsVDSCommand, log id: 267d9c2c >2013-03-10 20:10:00,256 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [4aa1184c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:10:00,257 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-13) [4aa1184c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 20:10:00,258 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [4aa1184c] Autorecovering 1 storage domains >2013-03-10 20:10:00,258 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [4aa1184c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 20:10:00,259 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [4da39129] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:10:00,259 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [4da39129] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 8:10 PM >2013-03-10 20:10:00,262 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [4da39129] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 8:10 PM >2013-03-10 20:10:02,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [1e536f05] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 66b6063e >2013-03-10 20:10:02,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [1e536f05] FINISH, GetHardwareInfoVDSCommand, log id: 66b6063e >2013-03-10 20:10:02,351 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-79) [5023a69c] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 20:10:02,354 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-79) [70309dbc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 20:10:02,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6d8fd392 >2013-03-10 20:10:08,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 20:10:08,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:10:08,359 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 20:10:08,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 6d8fd392 >2013-03-10 20:10:08,363 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [7a7a1525] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:10:08,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [7a7a1525] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6e4f8fef >2013-03-10 20:10:08,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [7a7a1525] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 20:10:08,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [7a7a1525] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6e4f8fef >2013-03-10 20:10:08,379 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [7a7a1525] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 20:10:08,380 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [70309dbc] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 20:10:08,389 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-79) [39cec4b0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:10:08,391 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [39cec4b0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 50f0f8ea >2013-03-10 20:10:08,392 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [39cec4b0] FINISH, SetVdsStatusVDSCommand, log id: 50f0f8ea >2013-03-10 20:10:08,398 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-79) [39cec4b0] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 20:10:08,426 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-79) [23ac04c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:10:08,429 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-79) [1fa5fcca] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:10:08,430 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-79) [1fa5fcca] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 20:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [2992ae79] Autorecovering 1 hosts >2013-03-10 20:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [2992ae79] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [7abee9e4] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [7abee9e4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [7abee9e4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1e1c5265 >2013-03-10 20:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [7abee9e4] FINISH, SetVdsStatusVDSCommand, log id: 1e1c5265 >2013-03-10 20:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [7abee9e4] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6962a680 >2013-03-10 20:15:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [7abee9e4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3de4c1ee >2013-03-10 20:15:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [7abee9e4] FINISH, GetHardwareInfoVDSCommand, log id: 3de4c1ee >2013-03-10 20:15:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-45) [2a5f1a83] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:15:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-45) [496b35d2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:15:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [496b35d2] FINISH, ActivateVdsVDSCommand, log id: 6962a680 >2013-03-10 20:15:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [496b35d2] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:15:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-45) [496b35d2] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 20:15:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [496b35d2] Autorecovering 1 storage domains >2013-03-10 20:15:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [496b35d2] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 20:15:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [77f9bcda] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:15:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [77f9bcda] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 8:15 PM >2013-03-10 20:15:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [77f9bcda] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 8:15 PM >2013-03-10 20:15:02,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [6675721] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 44fbd3bb >2013-03-10 20:15:02,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [6675721] FINISH, GetHardwareInfoVDSCommand, log id: 44fbd3bb >2013-03-10 20:15:02,715 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-21) [28436513] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 20:15:02,718 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-21) [38a829d6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 20:15:02,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 46fd1aa0 >2013-03-10 20:15:10,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 20:15:10,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:15:10,716 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 20:15:10,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 46fd1aa0 >2013-03-10 20:15:10,720 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [56809b6f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:15:10,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [56809b6f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 61c59123 >2013-03-10 20:15:10,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [56809b6f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 20:15:10,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [56809b6f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 61c59123 >2013-03-10 20:15:10,738 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [38a829d6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 20:15:10,748 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-21) [b29a2be] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:15:10,749 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [b29a2be] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5efa9931 >2013-03-10 20:15:10,750 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [b29a2be] FINISH, SetVdsStatusVDSCommand, log id: 5efa9931 >2013-03-10 20:15:10,756 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-21) [b29a2be] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 20:15:10,778 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-21) [794fe4b4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:15:10,780 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-21) [4fb169fa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:15:10,781 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-21) [4fb169fa] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 20:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [552c831b] Autorecovering 1 hosts >2013-03-10 20:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [552c831b] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [1a8c6454] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [1a8c6454] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:20:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [1a8c6454] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7ce7abdd >2013-03-10 20:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [1a8c6454] FINISH, SetVdsStatusVDSCommand, log id: 7ce7abdd >2013-03-10 20:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [1a8c6454] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 234faa3b >2013-03-10 20:20:00,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [1a8c6454] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 249a68e5 >2013-03-10 20:20:00,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [1a8c6454] FINISH, GetHardwareInfoVDSCommand, log id: 249a68e5 >2013-03-10 20:20:00,256 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [19ee39be] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:20:00,259 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [645c1aa6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:20:00,260 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [645c1aa6] FINISH, ActivateVdsVDSCommand, log id: 234faa3b >2013-03-10 20:20:00,279 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [645c1aa6] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:20:00,280 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [645c1aa6] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 20:20:00,281 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [645c1aa6] Autorecovering 1 storage domains >2013-03-10 20:20:00,281 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [645c1aa6] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 20:20:00,282 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [b10032] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:20:00,282 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [b10032] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 8:20 PM >2013-03-10 20:20:00,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [b10032] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 8:20 PM >2013-03-10 20:20:02,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [2a8077ba] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 58e831a3 >2013-03-10 20:20:02,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [2a8077ba] FINISH, GetHardwareInfoVDSCommand, log id: 58e831a3 >2013-03-10 20:20:02,058 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-96) [7d20bd2a] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 20:20:02,062 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-96) [39d7ed5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 20:20:02,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 67ff1f93 >2013-03-10 20:20:11,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 20:20:11,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:20:11,049 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 20:20:11,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 67ff1f93 >2013-03-10 20:20:11,052 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [1ef641db] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:20:11,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [1ef641db] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 8210582 >2013-03-10 20:20:11,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [1ef641db] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 20:20:11,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [1ef641db] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 8210582 >2013-03-10 20:20:11,068 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [1ef641db] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 20:20:11,068 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [39d7ed5] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 20:20:11,078 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-96) [4cbfc655] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:20:11,079 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [4cbfc655] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6ffd939b >2013-03-10 20:20:11,080 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [4cbfc655] FINISH, SetVdsStatusVDSCommand, log id: 6ffd939b >2013-03-10 20:20:11,086 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-96) [4cbfc655] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 20:20:11,110 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-96) [7ce59c65] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:20:11,113 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-96) [6651808c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:20:11,114 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-96) [6651808c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 20:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [2d03a309] Autorecovering 1 hosts >2013-03-10 20:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [2d03a309] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [4a9570e4] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [4a9570e4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:25:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [4a9570e4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4d4c462 >2013-03-10 20:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [4a9570e4] FINISH, SetVdsStatusVDSCommand, log id: 4d4c462 >2013-03-10 20:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-24) [4a9570e4] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 783ca2d1 >2013-03-10 20:25:00,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [4a9570e4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 125b37e7 >2013-03-10 20:25:00,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [4a9570e4] FINISH, GetHardwareInfoVDSCommand, log id: 125b37e7 >2013-03-10 20:25:00,234 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-24) [4b98df93] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:25:00,237 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-24) [7930cf47] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:25:00,238 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-24) [7930cf47] FINISH, ActivateVdsVDSCommand, log id: 783ca2d1 >2013-03-10 20:25:00,258 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [7930cf47] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:25:00,259 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-24) [7930cf47] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 20:25:00,260 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [7930cf47] Autorecovering 1 storage domains >2013-03-10 20:25:00,260 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [7930cf47] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 20:25:00,261 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [5cceb414] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:25:00,261 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [5cceb414] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 8:25 PM >2013-03-10 20:25:00,264 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [5cceb414] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 8:25 PM >2013-03-10 20:25:02,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [1d2fdfa2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3395c801 >2013-03-10 20:25:02,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [1d2fdfa2] FINISH, GetHardwareInfoVDSCommand, log id: 3395c801 >2013-03-10 20:25:02,404 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-76) [536922c8] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 20:25:02,407 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-76) [76dff535] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 20:25:02,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6143b5d9 >2013-03-10 20:25:10,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 20:25:10,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:25:10,405 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 20:25:10,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 6143b5d9 >2013-03-10 20:25:10,409 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [6b01b3ff] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:25:10,423 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [6b01b3ff] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 39aeffa7 >2013-03-10 20:25:10,423 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [6b01b3ff] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 20:25:10,423 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [6b01b3ff] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 39aeffa7 >2013-03-10 20:25:10,431 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [6b01b3ff] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 20:25:10,431 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [76dff535] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 20:25:10,441 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-76) [51859972] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:25:10,442 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [51859972] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1e6c9ac1 >2013-03-10 20:25:10,443 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [51859972] FINISH, SetVdsStatusVDSCommand, log id: 1e6c9ac1 >2013-03-10 20:25:10,448 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-76) [51859972] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 20:25:10,478 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-76) [62bbefa3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:25:10,481 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-76) [56f2646] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:25:10,482 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-76) [56f2646] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 20:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [3bff3564] Autorecovering 1 hosts >2013-03-10 20:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [3bff3564] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [7e9489a0] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [7e9489a0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [7e9489a0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1fe3472c >2013-03-10 20:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [7e9489a0] FINISH, SetVdsStatusVDSCommand, log id: 1fe3472c >2013-03-10 20:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-68) [7e9489a0] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3f73483a >2013-03-10 20:30:00,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [7e9489a0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 69ecfb3d >2013-03-10 20:30:00,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [7e9489a0] FINISH, GetHardwareInfoVDSCommand, log id: 69ecfb3d >2013-03-10 20:30:00,538 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-68) [393f76e3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:30:00,540 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-68) [3e3ebe8d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:30:00,542 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-68) [3e3ebe8d] FINISH, ActivateVdsVDSCommand, log id: 3f73483a >2013-03-10 20:30:00,561 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [3e3ebe8d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:30:00,562 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-68) [3e3ebe8d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 20:30:00,563 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [3e3ebe8d] Autorecovering 1 storage domains >2013-03-10 20:30:00,563 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [3e3ebe8d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 20:30:00,564 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [652270c0] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:30:00,564 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [652270c0] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 8:30 PM >2013-03-10 20:30:00,567 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [652270c0] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 8:30 PM >2013-03-10 20:30:01,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [71aefce4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3271b3a >2013-03-10 20:30:01,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [71aefce4] FINISH, GetHardwareInfoVDSCommand, log id: 3271b3a >2013-03-10 20:30:01,764 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-47) [728d963c] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 20:30:01,767 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-47) [4f106271] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 20:30:01,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5758e8f3 >2013-03-10 20:30:08,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 20:30:08,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:30:08,802 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 20:30:08,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 5758e8f3 >2013-03-10 20:30:08,806 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [50007429] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:30:08,824 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [50007429] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2f8d0cc1 >2013-03-10 20:30:08,824 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [50007429] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 20:30:08,824 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [50007429] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2f8d0cc1 >2013-03-10 20:30:08,826 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [50007429] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 20:30:08,827 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [4f106271] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 20:30:08,840 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-47) [509911a3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:30:08,841 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [509911a3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: c819105 >2013-03-10 20:30:08,842 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [509911a3] FINISH, SetVdsStatusVDSCommand, log id: c819105 >2013-03-10 20:30:08,849 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [509911a3] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 20:30:08,871 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [2c4f8ba7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:30:08,874 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [24449f3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:30:08,875 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-47) [24449f3] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 20:30:33,851 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-13) [4da39129] Start refreshing all users data >2013-03-10 20:30:35,007 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-22) [21b2cbb7] Attempting to update VMs/Templates Ovf. >2013-03-10 20:30:35,008 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-22) [21b2cbb7] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 20:30:35,009 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-22) [21b2cbb7] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 20:30:35,009 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-22) [21b2cbb7] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 20:30:35,009 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-22) [21b2cbb7] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 20:30:35,009 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-22) [21b2cbb7] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 20:30:35,010 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-22) [21b2cbb7] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 20:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-40) [39f1e06d] Autorecovering 1 hosts >2013-03-10 20:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-40) [39f1e06d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-40) [1b53e133] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-40) [1b53e133] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-40) [1b53e133] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 431d2c5f >2013-03-10 20:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-40) [1b53e133] FINISH, SetVdsStatusVDSCommand, log id: 431d2c5f >2013-03-10 20:35:00,021 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-40) [1b53e133] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6a3f9647 >2013-03-10 20:35:00,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-40) [1b53e133] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 56c6d6db >2013-03-10 20:35:00,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-40) [1b53e133] FINISH, GetHardwareInfoVDSCommand, log id: 56c6d6db >2013-03-10 20:35:00,106 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-40) [562ae5af] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:35:00,109 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-40) [505e6ad] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:35:00,110 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-40) [505e6ad] FINISH, ActivateVdsVDSCommand, log id: 6a3f9647 >2013-03-10 20:35:00,130 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-40) [505e6ad] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:35:00,131 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-40) [505e6ad] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 20:35:00,132 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-40) [505e6ad] Autorecovering 1 storage domains >2013-03-10 20:35:00,132 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-40) [505e6ad] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 20:35:00,133 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-40) [693c2ca8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:35:00,133 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-40) [693c2ca8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 8:35 PM >2013-03-10 20:35:00,135 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-40) [693c2ca8] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 8:35 PM >2013-03-10 20:35:00,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [764ac8be] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3a9b269b >2013-03-10 20:35:00,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [764ac8be] FINISH, GetHardwareInfoVDSCommand, log id: 3a9b269b >2013-03-10 20:35:00,210 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-37) [4e521f00] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 20:35:00,214 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-37) [535855f3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 20:35:00,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 36529f81 >2013-03-10 20:35:05,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 20:35:05,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:35:05,206 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 20:35:05,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 36529f81 >2013-03-10 20:35:05,210 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [2b97cb84] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:35:05,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [2b97cb84] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 158bf9b >2013-03-10 20:35:05,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [2b97cb84] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 20:35:05,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [2b97cb84] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 158bf9b >2013-03-10 20:35:05,227 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [2b97cb84] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 20:35:05,227 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [535855f3] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 20:35:05,237 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-37) [719d0dff] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:35:05,239 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [719d0dff] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4ff128a9 >2013-03-10 20:35:05,240 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [719d0dff] FINISH, SetVdsStatusVDSCommand, log id: 4ff128a9 >2013-03-10 20:35:05,245 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-37) [719d0dff] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 20:35:05,267 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-37) [5d6bca5e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:35:05,270 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-37) [6a2184f9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:35:05,271 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-37) [6a2184f9] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 20:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [6f93f931] Autorecovering 1 hosts >2013-03-10 20:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [6f93f931] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [cc62b78] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [cc62b78] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [cc62b78] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: c63c210 >2013-03-10 20:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [cc62b78] FINISH, SetVdsStatusVDSCommand, log id: c63c210 >2013-03-10 20:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-88) [cc62b78] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7bbc5722 >2013-03-10 20:40:00,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [cc62b78] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 33eef17b >2013-03-10 20:40:00,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [cc62b78] FINISH, GetHardwareInfoVDSCommand, log id: 33eef17b >2013-03-10 20:40:00,184 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-88) [3033c590] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:40:00,187 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-88) [7ba61c14] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:40:00,188 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-88) [7ba61c14] FINISH, ActivateVdsVDSCommand, log id: 7bbc5722 >2013-03-10 20:40:00,207 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [7ba61c14] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:40:00,208 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-88) [7ba61c14] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 20:40:00,210 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [7ba61c14] Autorecovering 1 storage domains >2013-03-10 20:40:00,210 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [7ba61c14] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 20:40:00,210 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [48b870e1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:40:00,210 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [48b870e1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 8:40 PM >2013-03-10 20:40:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [48b870e1] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 8:40 PM >2013-03-10 20:40:02,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [5cb70b17] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: dc8d4f9 >2013-03-10 20:40:02,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [5cb70b17] FINISH, GetHardwareInfoVDSCommand, log id: dc8d4f9 >2013-03-10 20:40:02,550 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-81) [7af59b3d] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 20:40:02,553 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-81) [7401ebbf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 20:40:02,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 50182742 >2013-03-10 20:40:07,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 20:40:07,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:40:07,539 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 20:40:07,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 50182742 >2013-03-10 20:40:07,543 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [7d0ca512] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:40:07,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [7d0ca512] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7459e242 >2013-03-10 20:40:07,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [7d0ca512] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 20:40:07,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [7d0ca512] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7459e242 >2013-03-10 20:40:07,559 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [7d0ca512] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 20:40:07,559 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [7401ebbf] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 20:40:07,568 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-81) [470584f0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:40:07,570 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [470584f0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3362a392 >2013-03-10 20:40:07,571 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [470584f0] FINISH, SetVdsStatusVDSCommand, log id: 3362a392 >2013-03-10 20:40:07,577 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-81) [470584f0] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 20:40:07,598 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-81) [3af9e991] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:40:07,601 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-81) [27d84c31] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:40:07,602 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-81) [27d84c31] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 20:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) [77351aaa] Autorecovering 1 hosts >2013-03-10 20:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) [77351aaa] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-97) [535d4742] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-97) [535d4742] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [535d4742] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1ffad398 >2013-03-10 20:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [535d4742] FINISH, SetVdsStatusVDSCommand, log id: 1ffad398 >2013-03-10 20:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-97) [535d4742] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: baee770 >2013-03-10 20:45:00,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [535d4742] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3a27aa97 >2013-03-10 20:45:00,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [535d4742] FINISH, GetHardwareInfoVDSCommand, log id: 3a27aa97 >2013-03-10 20:45:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-97) [6b37cd29] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:45:00,206 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-97) [772dd06e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:45:00,207 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-97) [772dd06e] FINISH, ActivateVdsVDSCommand, log id: baee770 >2013-03-10 20:45:00,227 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-97) [772dd06e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:45:00,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-97) [772dd06e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 20:45:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) [772dd06e] Autorecovering 1 storage domains >2013-03-10 20:45:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) [772dd06e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 20:45:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-97) [1435b523] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:45:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-97) [1435b523] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 8:45 PM >2013-03-10 20:45:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-97) [1435b523] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 8:45 PM >2013-03-10 20:45:01,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [17bcf0cf] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: f377396 >2013-03-10 20:45:01,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [17bcf0cf] FINISH, GetHardwareInfoVDSCommand, log id: f377396 >2013-03-10 20:45:01,908 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-92) [5877c6dd] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 20:45:01,911 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-92) [2a5859a4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 20:45:01,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5713bf2b >2013-03-10 20:45:05,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 20:45:05,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:45:05,895 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 20:45:05,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 5713bf2b >2013-03-10 20:45:05,898 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [7ad4b1bc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:45:05,912 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [7ad4b1bc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 162f267f >2013-03-10 20:45:05,912 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [7ad4b1bc] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 20:45:05,912 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [7ad4b1bc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 162f267f >2013-03-10 20:45:05,914 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [7ad4b1bc] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 20:45:05,914 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [2a5859a4] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 20:45:05,924 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-92) [64cc3577] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:45:05,925 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [64cc3577] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 407a3213 >2013-03-10 20:45:05,927 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [64cc3577] FINISH, SetVdsStatusVDSCommand, log id: 407a3213 >2013-03-10 20:45:05,933 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-92) [64cc3577] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 20:45:05,962 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-92) [73914256] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:45:05,965 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-92) [209f3120] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:45:05,966 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-92) [209f3120] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 20:50:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [2e09c15a] Autorecovering 1 hosts >2013-03-10 20:50:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [2e09c15a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:50:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [64bf042e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:50:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [64bf042e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [64bf042e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 336790f0 >2013-03-10 20:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [64bf042e] FINISH, SetVdsStatusVDSCommand, log id: 336790f0 >2013-03-10 20:50:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [64bf042e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7d47627e >2013-03-10 20:50:00,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [64bf042e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 398274d9 >2013-03-10 20:50:00,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [64bf042e] FINISH, GetHardwareInfoVDSCommand, log id: 398274d9 >2013-03-10 20:50:00,189 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-42) [67f013ee] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:50:00,192 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-42) [5e81f6d3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:50:00,193 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [5e81f6d3] FINISH, ActivateVdsVDSCommand, log id: 7d47627e >2013-03-10 20:50:00,216 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [5e81f6d3] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:50:00,217 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-42) [5e81f6d3] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 20:50:00,218 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [5e81f6d3] Autorecovering 1 storage domains >2013-03-10 20:50:00,218 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [5e81f6d3] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 20:50:00,219 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [1fdebb33] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:50:00,219 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [1fdebb33] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 8:50 PM >2013-03-10 20:50:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [1fdebb33] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 8:50 PM >2013-03-10 20:50:00,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-40) [693c2ca8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 11281d64 >2013-03-10 20:50:00,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-40) [693c2ca8] FINISH, GetHardwareInfoVDSCommand, log id: 11281d64 >2013-03-10 20:50:00,309 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-40) [700d8330] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 20:50:00,313 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-40) [3d58263] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 20:50:00,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2f02815d >2013-03-10 20:50:06,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 20:50:06,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:50:06,294 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 20:50:06,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 2f02815d >2013-03-10 20:50:06,299 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [62a7c33c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:50:06,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [62a7c33c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 42a3f6d8 >2013-03-10 20:50:06,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [62a7c33c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 20:50:06,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [62a7c33c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 42a3f6d8 >2013-03-10 20:50:06,319 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [62a7c33c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 20:50:06,319 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [3d58263] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 20:50:06,328 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-40) [bc6887e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:50:06,330 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-40) [bc6887e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 164203c8 >2013-03-10 20:50:06,331 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-40) [bc6887e] FINISH, SetVdsStatusVDSCommand, log id: 164203c8 >2013-03-10 20:50:06,337 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-40) [bc6887e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 20:50:06,359 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-40) [5e3b9438] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:50:06,362 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-40) [7a20673a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:50:06,363 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-40) [7a20673a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 20:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) [45887f55] Autorecovering 1 hosts >2013-03-10 20:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) [45887f55] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-85) [1285ea5d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-85) [1285ea5d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-85) [1285ea5d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 53b6ffd8 >2013-03-10 20:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-85) [1285ea5d] FINISH, SetVdsStatusVDSCommand, log id: 53b6ffd8 >2013-03-10 20:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-85) [1285ea5d] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1d085e10 >2013-03-10 20:55:00,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-85) [1285ea5d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 17cb47ee >2013-03-10 20:55:00,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-85) [1285ea5d] FINISH, GetHardwareInfoVDSCommand, log id: 17cb47ee >2013-03-10 20:55:00,184 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-85) [7c7ef35d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:55:00,187 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-85) [47e98643] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:55:00,188 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-85) [47e98643] FINISH, ActivateVdsVDSCommand, log id: 1d085e10 >2013-03-10 20:55:00,207 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-85) [47e98643] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 20:55:00,208 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-85) [47e98643] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 20:55:00,209 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) [47e98643] Autorecovering 1 storage domains >2013-03-10 20:55:00,209 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) [47e98643] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 20:55:00,210 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-85) [308d93ab] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:55:00,210 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-85) [308d93ab] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 8:55 PM >2013-03-10 20:55:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-85) [308d93ab] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 8:55 PM >2013-03-10 20:55:00,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [3b5c047] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 17ce38f9 >2013-03-10 20:55:00,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [3b5c047] FINISH, GetHardwareInfoVDSCommand, log id: 17ce38f9 >2013-03-10 20:55:00,650 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-72) [1418cd1d] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 20:55:00,654 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-72) [6b7cc034] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 20:55:00,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 55aecdb0 >2013-03-10 20:55:05,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 20:55:05,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 20:55:05,628 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 20:55:05,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 55aecdb0 >2013-03-10 20:55:05,631 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [9f21c4e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 20:55:05,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [9f21c4e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 715783a7 >2013-03-10 20:55:05,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [9f21c4e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 20:55:05,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [9f21c4e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 715783a7 >2013-03-10 20:55:05,648 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [9f21c4e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 20:55:05,648 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [6b7cc034] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 20:55:05,658 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-72) [1a52ea56] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:55:05,659 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [1a52ea56] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 73368511 >2013-03-10 20:55:05,660 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [1a52ea56] FINISH, SetVdsStatusVDSCommand, log id: 73368511 >2013-03-10 20:55:05,666 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [1a52ea56] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 20:55:05,689 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [5d3a624] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:55:05,691 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [104ef06f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 20:55:05,692 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-72) [104ef06f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 21:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [7ed08cc5] Autorecovering 1 hosts >2013-03-10 21:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [7ed08cc5] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [40b353d4] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [40b353d4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [40b353d4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: ff5dadb >2013-03-10 21:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [40b353d4] FINISH, SetVdsStatusVDSCommand, log id: ff5dadb >2013-03-10 21:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [40b353d4] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5a0806f0 >2013-03-10 21:00:00,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [40b353d4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6b07877e >2013-03-10 21:00:00,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [40b353d4] FINISH, GetHardwareInfoVDSCommand, log id: 6b07877e >2013-03-10 21:00:00,267 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-84) [783dc3a7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:00:00,270 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-84) [3f963f93] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:00:00,271 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [3f963f93] FINISH, ActivateVdsVDSCommand, log id: 5a0806f0 >2013-03-10 21:00:00,290 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [3f963f93] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:00:00,291 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-84) [3f963f93] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 21:00:00,292 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [3f963f93] Autorecovering 1 storage domains >2013-03-10 21:00:00,292 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [3f963f93] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 21:00:00,293 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [5288c5ba] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:00:00,293 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [5288c5ba] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 9:00 PM >2013-03-10 21:00:00,296 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [5288c5ba] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 9:00 PM >2013-03-10 21:00:02,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [10ce9d9a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 9032194 >2013-03-10 21:00:02,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [10ce9d9a] FINISH, GetHardwareInfoVDSCommand, log id: 9032194 >2013-03-10 21:00:02,981 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-100) [39961989] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 21:00:02,984 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-100) [2c1fecd7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 21:00:02,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 44082428 >2013-03-10 21:00:12,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 21:00:12,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:00:12,012 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 21:00:12,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 44082428 >2013-03-10 21:00:12,016 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [5ebd1ec7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:00:12,029 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [5ebd1ec7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 32e1f6cc >2013-03-10 21:00:12,029 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [5ebd1ec7] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 21:00:12,029 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [5ebd1ec7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 32e1f6cc >2013-03-10 21:00:12,032 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [5ebd1ec7] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 21:00:12,032 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [2c1fecd7] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 21:00:12,042 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-100) [490121a1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:00:12,044 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [490121a1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 59a21bf9 >2013-03-10 21:00:12,045 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [490121a1] FINISH, SetVdsStatusVDSCommand, log id: 59a21bf9 >2013-03-10 21:00:12,050 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-100) [490121a1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 21:00:12,072 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-100) [73a07379] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:00:12,074 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-100) [6a7362e8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:00:12,076 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-100) [6a7362e8] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 21:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [167e1326] Autorecovering 1 hosts >2013-03-10 21:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [167e1326] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [35cb7277] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [35cb7277] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [35cb7277] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 45a68844 >2013-03-10 21:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [35cb7277] FINISH, SetVdsStatusVDSCommand, log id: 45a68844 >2013-03-10 21:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-50) [35cb7277] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 27cf3cfa >2013-03-10 21:05:00,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [35cb7277] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 47e81b0f >2013-03-10 21:05:00,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [35cb7277] FINISH, GetHardwareInfoVDSCommand, log id: 47e81b0f >2013-03-10 21:05:00,243 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-50) [7af73ddc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:05:00,246 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-50) [172afb79] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:05:00,247 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-50) [172afb79] FINISH, ActivateVdsVDSCommand, log id: 27cf3cfa >2013-03-10 21:05:00,271 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [172afb79] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:05:00,272 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-50) [172afb79] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 21:05:00,273 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [172afb79] Autorecovering 1 storage domains >2013-03-10 21:05:00,273 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [172afb79] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 21:05:00,274 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [122bc673] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:05:00,274 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [122bc673] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 9:05 PM >2013-03-10 21:05:00,277 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [122bc673] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 9:05 PM >2013-03-10 21:05:00,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [2590f959] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2327be63 >2013-03-10 21:05:00,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [2590f959] FINISH, GetHardwareInfoVDSCommand, log id: 2327be63 >2013-03-10 21:05:00,355 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-12) [9be659c] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 21:05:00,359 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-12) [44880fd3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 21:05:00,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 73a99054 >2013-03-10 21:05:06,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 21:05:06,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:05:06,354 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 21:05:06,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 73a99054 >2013-03-10 21:05:06,358 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [79b739ae] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:05:06,371 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [79b739ae] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 50f0513f >2013-03-10 21:05:06,372 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [79b739ae] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 21:05:06,372 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [79b739ae] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 50f0513f >2013-03-10 21:05:06,374 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [79b739ae] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 21:05:06,374 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [44880fd3] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 21:05:06,384 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-12) [4b6d9fd6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:05:06,385 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [4b6d9fd6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3c8c5090 >2013-03-10 21:05:06,386 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [4b6d9fd6] FINISH, SetVdsStatusVDSCommand, log id: 3c8c5090 >2013-03-10 21:05:06,392 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-12) [4b6d9fd6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 21:05:06,414 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-12) [43cf44a1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:05:06,416 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-12) [1212d4d0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:05:06,418 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-12) [1212d4d0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 21:10:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [16379607] Autorecovering 1 hosts >2013-03-10 21:10:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [16379607] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [6e0d1a8a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [6e0d1a8a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [6e0d1a8a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1a1354a3 >2013-03-10 21:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [6e0d1a8a] FINISH, SetVdsStatusVDSCommand, log id: 1a1354a3 >2013-03-10 21:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [6e0d1a8a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5fe26aba >2013-03-10 21:10:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [6e0d1a8a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 16602389 >2013-03-10 21:10:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [6e0d1a8a] FINISH, GetHardwareInfoVDSCommand, log id: 16602389 >2013-03-10 21:10:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [734f0027] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:10:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [540a438] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:10:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [540a438] FINISH, ActivateVdsVDSCommand, log id: 5fe26aba >2013-03-10 21:10:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [540a438] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:10:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [540a438] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 21:10:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [540a438] Autorecovering 1 storage domains >2013-03-10 21:10:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [540a438] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 21:10:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [39635b34] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:10:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [39635b34] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 9:10 PM >2013-03-10 21:10:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [39635b34] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 9:10 PM >2013-03-10 21:10:00,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [7a58471c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 67da8956 >2013-03-10 21:10:00,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [7a58471c] FINISH, GetHardwareInfoVDSCommand, log id: 67da8956 >2013-03-10 21:10:00,704 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-59) [4545e7c5] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 21:10:00,707 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-59) [13969b4c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 21:10:00,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 220fa456 >2013-03-10 21:10:07,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 21:10:07,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:10:07,723 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 21:10:07,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 220fa456 >2013-03-10 21:10:07,727 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [5898cc2c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:10:07,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5898cc2c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 14ebc8fe >2013-03-10 21:10:07,741 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [5898cc2c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 21:10:07,741 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5898cc2c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 14ebc8fe >2013-03-10 21:10:07,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [5898cc2c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 21:10:07,743 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [13969b4c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 21:10:07,752 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-59) [6e0e464f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:10:07,754 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [6e0e464f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 675496ca >2013-03-10 21:10:07,755 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [6e0e464f] FINISH, SetVdsStatusVDSCommand, log id: 675496ca >2013-03-10 21:10:07,760 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [6e0e464f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 21:10:07,784 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [766827f0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:10:07,786 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [415ffe6a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:10:07,787 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-59) [415ffe6a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 21:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [4ce3d552] Autorecovering 1 hosts >2013-03-10 21:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [4ce3d552] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-91) [7e79cee3] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-91) [7e79cee3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [7e79cee3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5fbc3dc1 >2013-03-10 21:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [7e79cee3] FINISH, SetVdsStatusVDSCommand, log id: 5fbc3dc1 >2013-03-10 21:15:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-91) [7e79cee3] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 49871c30 >2013-03-10 21:15:00,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [7e79cee3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7d813de3 >2013-03-10 21:15:00,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [7e79cee3] FINISH, GetHardwareInfoVDSCommand, log id: 7d813de3 >2013-03-10 21:15:00,406 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-91) [1bbf2cec] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:15:00,409 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-91) [105ab721] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:15:00,410 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-91) [105ab721] FINISH, ActivateVdsVDSCommand, log id: 49871c30 >2013-03-10 21:15:00,429 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-91) [105ab721] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:15:00,430 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-91) [105ab721] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 21:15:00,431 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [105ab721] Autorecovering 1 storage domains >2013-03-10 21:15:00,431 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [105ab721] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 21:15:00,432 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-91) [7c1b7f3c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:15:00,432 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-91) [7c1b7f3c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 9:15 PM >2013-03-10 21:15:00,435 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-91) [7c1b7f3c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 9:15 PM >2013-03-10 21:15:02,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [5288c5ba] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5fc777f2 >2013-03-10 21:15:02,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [5288c5ba] FINISH, GetHardwareInfoVDSCommand, log id: 5fc777f2 >2013-03-10 21:15:02,068 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-84) [64dbe2a3] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 21:15:02,072 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-84) [1783568f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 21:15:02,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 67c86705 >2013-03-10 21:15:08,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 21:15:08,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:15:08,047 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 21:15:08,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 67c86705 >2013-03-10 21:15:08,051 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [1c065a43] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:15:08,067 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [1c065a43] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4dcecc52 >2013-03-10 21:15:08,067 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [1c065a43] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 21:15:08,067 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [1c065a43] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4dcecc52 >2013-03-10 21:15:08,069 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [1c065a43] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 21:15:08,069 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [1783568f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 21:15:08,081 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-84) [3f21467e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:15:08,082 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [3f21467e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 53d33a8b >2013-03-10 21:15:08,083 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [3f21467e] FINISH, SetVdsStatusVDSCommand, log id: 53d33a8b >2013-03-10 21:15:08,089 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-84) [3f21467e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 21:15:08,112 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-84) [33c89ef9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:15:08,115 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-84) [a9e9ef1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:15:08,116 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-84) [a9e9ef1] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 21:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [28f6d5ca] Autorecovering 1 hosts >2013-03-10 21:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [28f6d5ca] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-60) [1151f6a6] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-60) [1151f6a6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [1151f6a6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4a579583 >2013-03-10 21:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [1151f6a6] FINISH, SetVdsStatusVDSCommand, log id: 4a579583 >2013-03-10 21:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-60) [1151f6a6] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 19b1a4f8 >2013-03-10 21:20:00,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [1151f6a6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 65e6f538 >2013-03-10 21:20:00,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [1151f6a6] FINISH, GetHardwareInfoVDSCommand, log id: 65e6f538 >2013-03-10 21:20:00,416 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-60) [3b441dd1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:20:00,418 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-60) [450dd842] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:20:00,420 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-60) [450dd842] FINISH, ActivateVdsVDSCommand, log id: 19b1a4f8 >2013-03-10 21:20:00,440 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-60) [450dd842] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:20:00,441 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-60) [450dd842] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 21:20:00,442 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [450dd842] Autorecovering 1 storage domains >2013-03-10 21:20:00,442 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [450dd842] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 21:20:00,443 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-60) [75d8ecb1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:20:00,443 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-60) [75d8ecb1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 9:20 PM >2013-03-10 21:20:00,446 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-60) [75d8ecb1] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 9:20 PM >2013-03-10 21:20:02,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [122bc673] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4cdbb4f6 >2013-03-10 21:20:02,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [122bc673] FINISH, GetHardwareInfoVDSCommand, log id: 4cdbb4f6 >2013-03-10 21:20:02,421 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-50) [3aa8c3] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 21:20:02,425 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-50) [49808316] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 21:20:02,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 32cc39be >2013-03-10 21:20:07,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 21:20:07,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:20:07,427 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 21:20:07,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 32cc39be >2013-03-10 21:20:07,431 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [22dd045a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:20:07,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [22dd045a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 541ec1e0 >2013-03-10 21:20:07,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [22dd045a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 21:20:07,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [22dd045a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 541ec1e0 >2013-03-10 21:20:07,450 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [49808316] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 21:20:07,460 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-50) [6f3febed] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:20:07,461 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [6f3febed] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1863bdd4 >2013-03-10 21:20:07,462 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [6f3febed] FINISH, SetVdsStatusVDSCommand, log id: 1863bdd4 >2013-03-10 21:20:07,470 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-50) [6f3febed] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 21:20:07,493 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-50) [594cfd2f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:20:07,495 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-50) [9afc96e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:20:07,497 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-50) [9afc96e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 21:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [422e650f] Autorecovering 1 hosts >2013-03-10 21:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [422e650f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [b0026d2] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [b0026d2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [b0026d2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 133d0877 >2013-03-10 21:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [b0026d2] FINISH, SetVdsStatusVDSCommand, log id: 133d0877 >2013-03-10 21:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-18) [b0026d2] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 693ab47d >2013-03-10 21:25:00,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [b0026d2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5d5ad780 >2013-03-10 21:25:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [b0026d2] FINISH, GetHardwareInfoVDSCommand, log id: 5d5ad780 >2013-03-10 21:25:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [e8851e2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:25:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [1112eb24] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:25:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-18) [1112eb24] FINISH, ActivateVdsVDSCommand, log id: 693ab47d >2013-03-10 21:25:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [1112eb24] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:25:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [1112eb24] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 21:25:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [1112eb24] Autorecovering 1 storage domains >2013-03-10 21:25:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [1112eb24] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 21:25:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [4a1e14e4] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:25:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [4a1e14e4] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 9:25 PM >2013-03-10 21:25:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [4a1e14e4] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 9:25 PM >2013-03-10 21:25:01,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [39635b34] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4b626d74 >2013-03-10 21:25:01,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [39635b34] FINISH, GetHardwareInfoVDSCommand, log id: 4b626d74 >2013-03-10 21:25:01,782 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-36) [2ade46dd] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 21:25:01,786 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-36) [232053e0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 21:25:01,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 62da62d6 >2013-03-10 21:25:07,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 21:25:07,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:25:07,080 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 21:25:07,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 62da62d6 >2013-03-10 21:25:07,084 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [4fb5db33] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:25:07,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4fb5db33] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3e3689d8 >2013-03-10 21:25:07,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [4fb5db33] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 21:25:07,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4fb5db33] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3e3689d8 >2013-03-10 21:25:07,100 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [4fb5db33] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 21:25:07,101 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [232053e0] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 21:25:07,111 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-36) [918908b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:25:07,113 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [918908b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5a86b3f0 >2013-03-10 21:25:07,114 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [918908b] FINISH, SetVdsStatusVDSCommand, log id: 5a86b3f0 >2013-03-10 21:25:07,120 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [918908b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 21:25:07,143 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [4a97bae0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:25:07,147 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [361df729] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:25:07,148 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-36) [361df729] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 21:30:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [2f45233e] Autorecovering 1 hosts >2013-03-10 21:30:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [2f45233e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [5a0736a5] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [5a0736a5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [5a0736a5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 19557863 >2013-03-10 21:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [5a0736a5] FINISH, SetVdsStatusVDSCommand, log id: 19557863 >2013-03-10 21:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [5a0736a5] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4144034c >2013-03-10 21:30:00,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [5a0736a5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 609339b6 >2013-03-10 21:30:00,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [5a0736a5] FINISH, GetHardwareInfoVDSCommand, log id: 609339b6 >2013-03-10 21:30:00,410 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-69) [639112a8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:30:00,413 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-69) [6583d5b3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:30:00,414 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [6583d5b3] FINISH, ActivateVdsVDSCommand, log id: 4144034c >2013-03-10 21:30:00,433 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [6583d5b3] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:30:00,434 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-69) [6583d5b3] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 21:30:00,435 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [6583d5b3] Autorecovering 1 storage domains >2013-03-10 21:30:00,435 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [6583d5b3] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 21:30:00,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [60227064] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:30:00,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [60227064] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 9:30 PM >2013-03-10 21:30:00,439 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [60227064] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 9:30 PM >2013-03-10 21:30:01,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-6) [5e4b1f92] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7c828f5c >2013-03-10 21:30:01,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-6) [5e4b1f92] FINISH, GetHardwareInfoVDSCommand, log id: 7c828f5c >2013-03-10 21:30:01,435 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-6) [1854c06d] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 21:30:01,438 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-6) [7cfb549f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 21:30:01,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 459f5ec9 >2013-03-10 21:30:08,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 21:30:08,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:30:08,492 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 21:30:08,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 459f5ec9 >2013-03-10 21:30:08,496 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [711e0d77] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:30:08,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [711e0d77] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 22918215 >2013-03-10 21:30:08,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [711e0d77] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 21:30:08,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [711e0d77] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 22918215 >2013-03-10 21:30:08,513 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [711e0d77] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 21:30:08,513 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [7cfb549f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 21:30:08,523 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-6) [156e2ce2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:30:08,524 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-6) [156e2ce2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 48542975 >2013-03-10 21:30:08,525 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-6) [156e2ce2] FINISH, SetVdsStatusVDSCommand, log id: 48542975 >2013-03-10 21:30:08,531 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-6) [156e2ce2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 21:30:08,558 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-6) [1fc55455] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:30:08,561 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-6) [cbde079] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:30:08,562 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-6) [cbde079] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 21:30:33,853 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-85) [308d93ab] Start refreshing all users data >2013-03-10 21:30:35,010 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) [361df729] Attempting to update VMs/Templates Ovf. >2013-03-10 21:30:35,011 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) [361df729] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 21:30:35,012 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) [361df729] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 21:30:35,012 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) [361df729] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 21:30:35,013 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) [361df729] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 21:30:35,013 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) [361df729] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 21:30:35,013 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) [361df729] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 21:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [31e5ebee] Autorecovering 1 hosts >2013-03-10 21:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [31e5ebee] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [676fabc3] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [676fabc3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [676fabc3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 38a5fa2f >2013-03-10 21:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [676fabc3] FINISH, SetVdsStatusVDSCommand, log id: 38a5fa2f >2013-03-10 21:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-90) [676fabc3] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 216d2f47 >2013-03-10 21:35:00,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [676fabc3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1f7de7fe >2013-03-10 21:35:00,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [676fabc3] FINISH, GetHardwareInfoVDSCommand, log id: 1f7de7fe >2013-03-10 21:35:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-90) [2af31043] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:35:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-90) [474efdec] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:35:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-90) [474efdec] FINISH, ActivateVdsVDSCommand, log id: 216d2f47 >2013-03-10 21:35:00,219 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [474efdec] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:35:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [474efdec] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 21:35:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [474efdec] Autorecovering 1 storage domains >2013-03-10 21:35:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [474efdec] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 21:35:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [40525bec] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:35:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [40525bec] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 9:35 PM >2013-03-10 21:35:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [40525bec] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 9:35 PM >2013-03-10 21:35:02,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [1fdebb33] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 434d094b >2013-03-10 21:35:02,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [1fdebb33] FINISH, GetHardwareInfoVDSCommand, log id: 434d094b >2013-03-10 21:35:02,846 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-42) [193d375a] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 21:35:02,849 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-42) [618cd1d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 21:35:02,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: e38274f >2013-03-10 21:35:11,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 21:35:11,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:35:11,839 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 21:35:11,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: e38274f >2013-03-10 21:35:11,843 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [118b7e46] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:35:11,856 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [118b7e46] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7ed6152d >2013-03-10 21:35:11,856 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [118b7e46] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 21:35:11,856 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [118b7e46] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7ed6152d >2013-03-10 21:35:11,858 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [118b7e46] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 21:35:11,859 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [618cd1d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 21:35:11,868 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-42) [2b24f335] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:35:11,869 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [2b24f335] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 19b28c4f >2013-03-10 21:35:11,870 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [2b24f335] FINISH, SetVdsStatusVDSCommand, log id: 19b28c4f >2013-03-10 21:35:11,876 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-42) [2b24f335] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 21:35:11,898 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-42) [2b4c3b82] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:35:11,901 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-42) [26f6c0fe] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:35:11,902 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-42) [26f6c0fe] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 21:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [372fae9f] Autorecovering 1 hosts >2013-03-10 21:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [372fae9f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [4b1d3e64] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [4b1d3e64] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [4b1d3e64] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4f2e21ba >2013-03-10 21:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [4b1d3e64] FINISH, SetVdsStatusVDSCommand, log id: 4f2e21ba >2013-03-10 21:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-20) [4b1d3e64] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 38ed6851 >2013-03-10 21:40:00,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [4b1d3e64] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5fca26bc >2013-03-10 21:40:00,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [4b1d3e64] FINISH, GetHardwareInfoVDSCommand, log id: 5fca26bc >2013-03-10 21:40:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-20) [655cb9b1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:40:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-20) [440a9291] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:40:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-20) [440a9291] FINISH, ActivateVdsVDSCommand, log id: 38ed6851 >2013-03-10 21:40:00,224 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [440a9291] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:40:00,225 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-20) [440a9291] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 21:40:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [440a9291] Autorecovering 1 storage domains >2013-03-10 21:40:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [440a9291] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 21:40:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [2bf4b7c5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:40:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [2bf4b7c5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 9:40 PM >2013-03-10 21:40:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [2bf4b7c5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 9:40 PM >2013-03-10 21:40:00,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [1bed0da9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2b1d13f7 >2013-03-10 21:40:00,267 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [1bed0da9] FINISH, GetHardwareInfoVDSCommand, log id: 2b1d13f7 >2013-03-10 21:40:00,303 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-87) [d0cf491] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 21:40:00,307 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-87) [4335c1ee] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 21:40:00,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: c9d5a2b >2013-03-10 21:40:06,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 21:40:06,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:40:06,319 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 21:40:06,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: c9d5a2b >2013-03-10 21:40:06,323 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [4195c94b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:40:06,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [4195c94b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 22cb2090 >2013-03-10 21:40:06,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [4195c94b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 21:40:06,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [4195c94b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 22cb2090 >2013-03-10 21:40:06,340 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [4195c94b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 21:40:06,340 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [4335c1ee] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 21:40:06,350 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-87) [20131354] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:40:06,351 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [20131354] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: baf0cce >2013-03-10 21:40:06,352 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [20131354] FINISH, SetVdsStatusVDSCommand, log id: baf0cce >2013-03-10 21:40:06,358 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [20131354] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 21:40:06,380 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [299c6de5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:40:06,382 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [2b6a4589] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:40:06,383 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-87) [2b6a4589] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 21:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [54ec0185] Autorecovering 1 hosts >2013-03-10 21:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [54ec0185] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-4) [61cc180a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-4) [61cc180a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [61cc180a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6fe33a96 >2013-03-10 21:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [61cc180a] FINISH, SetVdsStatusVDSCommand, log id: 6fe33a96 >2013-03-10 21:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-4) [61cc180a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 40fdcc82 >2013-03-10 21:45:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [61cc180a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 22a6cd83 >2013-03-10 21:45:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [61cc180a] FINISH, GetHardwareInfoVDSCommand, log id: 22a6cd83 >2013-03-10 21:45:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-4) [58c8aeb6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:45:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-4) [75f6f1ac] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:45:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-4) [75f6f1ac] FINISH, ActivateVdsVDSCommand, log id: 40fdcc82 >2013-03-10 21:45:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-4) [75f6f1ac] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:45:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-4) [75f6f1ac] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 21:45:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [75f6f1ac] Autorecovering 1 storage domains >2013-03-10 21:45:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [75f6f1ac] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 21:45:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-4) [30e5b0bc] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:45:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-4) [30e5b0bc] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 9:45 PM >2013-03-10 21:45:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-4) [30e5b0bc] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 9:45 PM >2013-03-10 21:45:00,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [77078b90] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1b1e3d6b >2013-03-10 21:45:00,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [77078b90] FINISH, GetHardwareInfoVDSCommand, log id: 1b1e3d6b >2013-03-10 21:45:00,687 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-55) [1905b170] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 21:45:00,691 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-55) [795ffd0a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 21:45:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 129deb66 >2013-03-10 21:45:06,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 21:45:06,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:45:06,675 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 21:45:06,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 129deb66 >2013-03-10 21:45:06,679 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [143b9e95] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:45:06,693 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [143b9e95] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 21c8e2b0 >2013-03-10 21:45:06,693 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [143b9e95] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 21:45:06,693 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [143b9e95] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 21c8e2b0 >2013-03-10 21:45:06,695 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [143b9e95] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 21:45:06,696 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [795ffd0a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 21:45:06,705 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-55) [303fffca] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:45:06,706 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [303fffca] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: f1bb159 >2013-03-10 21:45:06,708 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [303fffca] FINISH, SetVdsStatusVDSCommand, log id: f1bb159 >2013-03-10 21:45:06,713 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-55) [303fffca] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 21:45:06,734 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-55) [36c41a18] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:45:06,737 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-55) [1c38d809] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:45:06,738 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-55) [1c38d809] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 21:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [52a39daf] Autorecovering 1 hosts >2013-03-10 21:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [52a39daf] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [367852b3] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [367852b3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [367852b3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 59465c05 >2013-03-10 21:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [367852b3] FINISH, SetVdsStatusVDSCommand, log id: 59465c05 >2013-03-10 21:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-77) [367852b3] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: e938dcb >2013-03-10 21:50:00,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [367852b3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 47983e33 >2013-03-10 21:50:00,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [367852b3] FINISH, GetHardwareInfoVDSCommand, log id: 47983e33 >2013-03-10 21:50:00,482 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-77) [1a73da00] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:50:00,485 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-77) [59fa8313] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:50:00,486 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-77) [59fa8313] FINISH, ActivateVdsVDSCommand, log id: e938dcb >2013-03-10 21:50:00,506 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [59fa8313] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:50:00,507 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-77) [59fa8313] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 21:50:00,508 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [59fa8313] Autorecovering 1 storage domains >2013-03-10 21:50:00,508 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [59fa8313] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 21:50:00,509 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [6aa6891a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:50:00,509 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [6aa6891a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 9:50 PM >2013-03-10 21:50:00,512 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [6aa6891a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 9:50 PM >2013-03-10 21:50:00,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [73847da4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 61eadca8 >2013-03-10 21:50:00,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [73847da4] FINISH, GetHardwareInfoVDSCommand, log id: 61eadca8 >2013-03-10 21:50:01,022 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-54) [2695e3fd] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 21:50:01,025 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-54) [344832ed] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 21:50:01,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7c2c0f3c >2013-03-10 21:50:08,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 21:50:08,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:50:08,002 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 21:50:08,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 7c2c0f3c >2013-03-10 21:50:08,006 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [3fb87990] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:50:08,020 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [3fb87990] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 51021135 >2013-03-10 21:50:08,020 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [3fb87990] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 21:50:08,020 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [3fb87990] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 51021135 >2013-03-10 21:50:08,022 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [3fb87990] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 21:50:08,022 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [344832ed] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 21:50:08,031 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-54) [3a9f1e00] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:50:08,033 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [3a9f1e00] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 115c8e80 >2013-03-10 21:50:08,034 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [3a9f1e00] FINISH, SetVdsStatusVDSCommand, log id: 115c8e80 >2013-03-10 21:50:08,040 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [3a9f1e00] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 21:50:08,063 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [3aea9fdf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:50:08,066 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [413384bd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:50:08,067 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-54) [413384bd] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 21:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [7e0264f3] Autorecovering 1 hosts >2013-03-10 21:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [7e0264f3] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-74) [290004a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-74) [290004a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [290004a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 543b6d48 >2013-03-10 21:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [290004a] FINISH, SetVdsStatusVDSCommand, log id: 543b6d48 >2013-03-10 21:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-74) [290004a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 283bedb9 >2013-03-10 21:55:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [290004a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 27329205 >2013-03-10 21:55:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [290004a] FINISH, GetHardwareInfoVDSCommand, log id: 27329205 >2013-03-10 21:55:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-74) [3dc753d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:55:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-74) [4a393507] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:55:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-74) [4a393507] FINISH, ActivateVdsVDSCommand, log id: 283bedb9 >2013-03-10 21:55:00,111 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-74) [4a393507] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 21:55:00,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-74) [4a393507] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 21:55:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [4a393507] Autorecovering 1 storage domains >2013-03-10 21:55:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [4a393507] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 21:55:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-74) [3d67cecb] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:55:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-74) [3d67cecb] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 9:55 PM >2013-03-10 21:55:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-74) [3d67cecb] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 9:55 PM >2013-03-10 21:55:02,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [553c3ff2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 251bd97b >2013-03-10 21:55:02,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [553c3ff2] FINISH, GetHardwareInfoVDSCommand, log id: 251bd97b >2013-03-10 21:55:02,348 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-78) [23cb63a7] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 21:55:02,351 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-78) [5940e3cd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 21:55:02,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 17df24b2 >2013-03-10 21:55:09,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 21:55:09,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 21:55:09,331 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 21:55:09,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 17df24b2 >2013-03-10 21:55:09,334 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [53e5f53e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 21:55:09,348 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [53e5f53e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2bc23228 >2013-03-10 21:55:09,348 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [53e5f53e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 21:55:09,348 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [53e5f53e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2bc23228 >2013-03-10 21:55:09,350 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [53e5f53e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 21:55:09,351 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [5940e3cd] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 21:55:09,360 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-78) [14d84f43] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:55:09,362 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [14d84f43] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2d1966cd >2013-03-10 21:55:09,363 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [14d84f43] FINISH, SetVdsStatusVDSCommand, log id: 2d1966cd >2013-03-10 21:55:09,368 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-78) [14d84f43] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 21:55:09,391 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-78) [465b64ef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:55:09,393 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-78) [2baffae0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 21:55:09,394 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-78) [2baffae0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 22:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [7998d983] Autorecovering 1 hosts >2013-03-10 22:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [7998d983] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [76a3ff2c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [76a3ff2c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [76a3ff2c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 658cf431 >2013-03-10 22:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [76a3ff2c] FINISH, SetVdsStatusVDSCommand, log id: 658cf431 >2013-03-10 22:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [76a3ff2c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 695d803 >2013-03-10 22:00:00,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [76a3ff2c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6779a3d3 >2013-03-10 22:00:00,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [76a3ff2c] FINISH, GetHardwareInfoVDSCommand, log id: 6779a3d3 >2013-03-10 22:00:00,101 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [5c6325ac] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:00:00,104 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [4c436b1c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:00:00,105 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [4c436b1c] FINISH, ActivateVdsVDSCommand, log id: 695d803 >2013-03-10 22:00:00,124 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [4c436b1c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:00:00,125 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-52) [4c436b1c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 22:00:00,126 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [4c436b1c] Autorecovering 1 storage domains >2013-03-10 22:00:00,126 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [4c436b1c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 22:00:00,127 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [719e370d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:00:00,127 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [719e370d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 10:00 PM >2013-03-10 22:00:00,130 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [719e370d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 10:00 PM >2013-03-10 22:00:00,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [66ca51fd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2c3615ec >2013-03-10 22:00:00,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [66ca51fd] FINISH, GetHardwareInfoVDSCommand, log id: 2c3615ec >2013-03-10 22:00:00,682 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-86) [25bd19a8] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 22:00:00,685 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-86) [7995458b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 22:00:00,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 71a4656e >2013-03-10 22:00:08,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 22:00:08,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:00:08,743 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 22:00:08,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 71a4656e >2013-03-10 22:00:08,746 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [88aeccf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:00:08,761 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [88aeccf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 48978543 >2013-03-10 22:00:08,761 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [88aeccf] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 22:00:08,761 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [88aeccf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 48978543 >2013-03-10 22:00:08,763 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [88aeccf] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 22:00:08,764 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [7995458b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 22:00:08,773 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-86) [15c20c5b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:00:08,774 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [15c20c5b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1f03a9f7 >2013-03-10 22:00:08,775 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [15c20c5b] FINISH, SetVdsStatusVDSCommand, log id: 1f03a9f7 >2013-03-10 22:00:08,781 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-86) [15c20c5b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 22:00:08,802 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-86) [468daee8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:00:08,805 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-86) [6f06a147] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:00:08,806 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-86) [6f06a147] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 22:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [7e74c320] Autorecovering 1 hosts >2013-03-10 22:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [7e74c320] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [6a3d5e80] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [6a3d5e80] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [6a3d5e80] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6d5d8413 >2013-03-10 22:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [6a3d5e80] FINISH, SetVdsStatusVDSCommand, log id: 6d5d8413 >2013-03-10 22:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-89) [6a3d5e80] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 62750067 >2013-03-10 22:05:00,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [6a3d5e80] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 52cc6f4e >2013-03-10 22:05:00,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [6a3d5e80] FINISH, GetHardwareInfoVDSCommand, log id: 52cc6f4e >2013-03-10 22:05:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-89) [6f2abd3f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:05:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-89) [46d89efd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:05:00,205 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-89) [46d89efd] FINISH, ActivateVdsVDSCommand, log id: 62750067 >2013-03-10 22:05:00,224 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [46d89efd] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:05:00,225 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-89) [46d89efd] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 22:05:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [46d89efd] Autorecovering 1 storage domains >2013-03-10 22:05:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [46d89efd] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 22:05:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [353b2ed0] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:05:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [353b2ed0] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 10:05 PM >2013-03-10 22:05:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [353b2ed0] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 10:05 PM >2013-03-10 22:05:03,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [75d8ecb1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 325569a3 >2013-03-10 22:05:03,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [75d8ecb1] FINISH, GetHardwareInfoVDSCommand, log id: 325569a3 >2013-03-10 22:05:03,088 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-60) [4fc6accf] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 22:05:03,092 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-60) [6f651a2e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 22:05:03,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3ae887a9 >2013-03-10 22:05:11,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 22:05:11,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:05:11,092 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 22:05:11,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 3ae887a9 >2013-03-10 22:05:11,096 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [1ffbfad] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:05:11,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [1ffbfad] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7b4f3dee >2013-03-10 22:05:11,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [1ffbfad] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 22:05:11,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [1ffbfad] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7b4f3dee >2013-03-10 22:05:11,111 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [1ffbfad] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 22:05:11,112 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [6f651a2e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 22:05:11,122 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-60) [5c2f7b96] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:05:11,123 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [5c2f7b96] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7b6e65c0 >2013-03-10 22:05:11,124 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [5c2f7b96] FINISH, SetVdsStatusVDSCommand, log id: 7b6e65c0 >2013-03-10 22:05:11,130 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-60) [5c2f7b96] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 22:05:11,154 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-60) [18289d83] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:05:11,156 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-60) [64cdd242] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:05:11,157 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-60) [64cdd242] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 22:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) [626594b9] Autorecovering 1 hosts >2013-03-10 22:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) [626594b9] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-34) [7fa01ce5] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-34) [7fa01ce5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [7fa01ce5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: d21d3b9 >2013-03-10 22:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [7fa01ce5] FINISH, SetVdsStatusVDSCommand, log id: d21d3b9 >2013-03-10 22:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-34) [7fa01ce5] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 65f839fa >2013-03-10 22:10:00,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [7fa01ce5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3dd0833a >2013-03-10 22:10:00,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [7fa01ce5] FINISH, GetHardwareInfoVDSCommand, log id: 3dd0833a >2013-03-10 22:10:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-34) [42c39feb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:10:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-34) [3f294705] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:10:00,206 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-34) [3f294705] FINISH, ActivateVdsVDSCommand, log id: 65f839fa >2013-03-10 22:10:00,225 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-34) [3f294705] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:10:00,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-34) [3f294705] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 22:10:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) [3f294705] Autorecovering 1 storage domains >2013-03-10 22:10:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-34) [3f294705] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 22:10:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-34) [27c0d33b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:10:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-34) [27c0d33b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 10:10 PM >2013-03-10 22:10:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-34) [27c0d33b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 10:10 PM >2013-03-10 22:10:02,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [2f79528e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2d4ae1a1 >2013-03-10 22:10:02,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [2f79528e] FINISH, GetHardwareInfoVDSCommand, log id: 2d4ae1a1 >2013-03-10 22:10:02,442 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-16) [248588ce] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 22:10:02,446 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-16) [42786e7c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 22:10:02,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 39b91d8c >2013-03-10 22:10:07,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 22:10:07,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:10:07,438 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 22:10:07,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 39b91d8c >2013-03-10 22:10:07,442 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [cf3856c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:10:07,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [cf3856c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 11cee0d6 >2013-03-10 22:10:07,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [cf3856c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 22:10:07,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [cf3856c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 11cee0d6 >2013-03-10 22:10:07,464 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [cf3856c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 22:10:07,465 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [42786e7c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 22:10:07,474 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-16) [797b2c0e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:10:07,475 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [797b2c0e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2d19297c >2013-03-10 22:10:07,477 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [797b2c0e] FINISH, SetVdsStatusVDSCommand, log id: 2d19297c >2013-03-10 22:10:07,483 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-16) [797b2c0e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 22:10:07,506 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-16) [3fa8078b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:10:07,508 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-16) [4e3ff719] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:10:07,509 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-16) [4e3ff719] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 22:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [37b009bb] Autorecovering 1 hosts >2013-03-10 22:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [37b009bb] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [418bee51] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [418bee51] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [418bee51] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 717b2d9c >2013-03-10 22:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [418bee51] FINISH, SetVdsStatusVDSCommand, log id: 717b2d9c >2013-03-10 22:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [418bee51] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5fe946d >2013-03-10 22:15:00,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [418bee51] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3abc1290 >2013-03-10 22:15:00,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [418bee51] FINISH, GetHardwareInfoVDSCommand, log id: 3abc1290 >2013-03-10 22:15:00,191 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-14) [71ed3440] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:15:00,194 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-14) [629c0e32] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:15:00,195 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [629c0e32] FINISH, ActivateVdsVDSCommand, log id: 5fe946d >2013-03-10 22:15:00,214 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [629c0e32] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:15:00,215 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-14) [629c0e32] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 22:15:00,217 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [629c0e32] Autorecovering 1 storage domains >2013-03-10 22:15:00,217 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [629c0e32] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 22:15:00,217 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [68ad8fa8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:15:00,217 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [68ad8fa8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 10:15 PM >2013-03-10 22:15:00,220 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [68ad8fa8] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 10:15 PM >2013-03-10 22:15:01,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [7702598c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 197c0b94 >2013-03-10 22:15:01,751 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [7702598c] FINISH, GetHardwareInfoVDSCommand, log id: 197c0b94 >2013-03-10 22:15:01,788 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-46) [478b55b0] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 22:15:01,792 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-46) [36d8c086] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 22:15:01,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: be901cd >2013-03-10 22:15:10,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 22:15:10,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:15:10,101 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 22:15:10,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: be901cd >2013-03-10 22:15:10,105 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [4fa9d7fe] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:15:10,119 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4fa9d7fe] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1bbd2e68 >2013-03-10 22:15:10,119 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [4fa9d7fe] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 22:15:10,119 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4fa9d7fe] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1bbd2e68 >2013-03-10 22:15:10,121 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [4fa9d7fe] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 22:15:10,122 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [36d8c086] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 22:15:10,131 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-46) [4669b619] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:15:10,133 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [4669b619] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3b4aa8a5 >2013-03-10 22:15:10,134 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [4669b619] FINISH, SetVdsStatusVDSCommand, log id: 3b4aa8a5 >2013-03-10 22:15:10,140 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-46) [4669b619] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 22:15:10,161 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-46) [315c2d11] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:15:10,164 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-46) [2201893f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:15:10,165 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-46) [2201893f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 22:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [351681a3] Autorecovering 1 hosts >2013-03-10 22:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [351681a3] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [2492ca32] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [2492ca32] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [2492ca32] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7fb8464 >2013-03-10 22:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [2492ca32] FINISH, SetVdsStatusVDSCommand, log id: 7fb8464 >2013-03-10 22:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [2492ca32] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7481a74c >2013-03-10 22:20:01,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [2492ca32] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 449951f4 >2013-03-10 22:20:01,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [2492ca32] FINISH, GetHardwareInfoVDSCommand, log id: 449951f4 >2013-03-10 22:20:01,058 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [261e855] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:20:01,060 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [7e65c404] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:20:01,062 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [7e65c404] FINISH, ActivateVdsVDSCommand, log id: 7481a74c >2013-03-10 22:20:01,081 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [7e65c404] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:20:01,082 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-94) [7e65c404] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 22:20:01,083 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [7e65c404] Autorecovering 1 storage domains >2013-03-10 22:20:01,083 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [7e65c404] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 22:20:01,084 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [7a32b6f5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:20:01,084 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [7a32b6f5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 10:20 PM >2013-03-10 22:20:01,087 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [7a32b6f5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 10:20 PM >2013-03-10 22:20:01,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [35f834a6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7c78f9d5 >2013-03-10 22:20:01,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [35f834a6] FINISH, GetHardwareInfoVDSCommand, log id: 7c78f9d5 >2013-03-10 22:20:01,439 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-80) [6f35d8af] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 22:20:01,442 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-80) [4dcffd17] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 22:20:01,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 56effae2 >2013-03-10 22:20:07,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 22:20:07,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:20:07,432 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 22:20:07,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 56effae2 >2013-03-10 22:20:07,435 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [15b5646] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:20:07,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [15b5646] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 11a580f1 >2013-03-10 22:20:07,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [15b5646] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 22:20:07,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [15b5646] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 11a580f1 >2013-03-10 22:20:07,454 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [4dcffd17] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 22:20:07,463 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-80) [aeefdda] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:20:07,465 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [aeefdda] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2e5cef20 >2013-03-10 22:20:07,466 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [aeefdda] FINISH, SetVdsStatusVDSCommand, log id: 2e5cef20 >2013-03-10 22:20:07,472 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-80) [aeefdda] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 22:20:07,493 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-80) [58b16413] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:20:07,496 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-80) [e4e30f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:20:07,497 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-80) [e4e30f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 22:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [547b8320] Autorecovering 1 hosts >2013-03-10 22:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [547b8320] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [6e6148ff] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [6e6148ff] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:25:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [6e6148ff] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5ee809ae >2013-03-10 22:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [6e6148ff] FINISH, SetVdsStatusVDSCommand, log id: 5ee809ae >2013-03-10 22:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [6e6148ff] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 290aaeef >2013-03-10 22:25:00,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [6e6148ff] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 62df76f1 >2013-03-10 22:25:00,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [6e6148ff] FINISH, GetHardwareInfoVDSCommand, log id: 62df76f1 >2013-03-10 22:25:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [4f5f16a7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:25:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [74032fbd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:25:00,094 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [74032fbd] FINISH, ActivateVdsVDSCommand, log id: 290aaeef >2013-03-10 22:25:00,113 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [74032fbd] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:25:00,114 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [74032fbd] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 22:25:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [74032fbd] Autorecovering 1 storage domains >2013-03-10 22:25:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [74032fbd] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 22:25:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [4401fa88] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:25:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [4401fa88] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 10:25 PM >2013-03-10 22:25:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [4401fa88] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 10:25 PM >2013-03-10 22:25:01,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [929b8aa] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 9630aea >2013-03-10 22:25:01,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [929b8aa] FINISH, GetHardwareInfoVDSCommand, log id: 9630aea >2013-03-10 22:25:01,782 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-64) [63a0ede5] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 22:25:01,785 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-64) [5c12c64e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 22:25:01,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 788c2df4 >2013-03-10 22:25:08,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 22:25:08,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:25:08,767 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 22:25:08,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 788c2df4 >2013-03-10 22:25:08,770 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [7bcab1b0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:25:08,784 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [7bcab1b0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 43446a14 >2013-03-10 22:25:08,784 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [7bcab1b0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 22:25:08,785 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [7bcab1b0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 43446a14 >2013-03-10 22:25:08,787 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [7bcab1b0] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 22:25:08,787 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [5c12c64e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 22:25:08,796 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-64) [2c695fb1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:25:08,798 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [2c695fb1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6eb55f36 >2013-03-10 22:25:08,799 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [2c695fb1] FINISH, SetVdsStatusVDSCommand, log id: 6eb55f36 >2013-03-10 22:25:08,805 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-64) [2c695fb1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 22:25:08,827 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-64) [30ef4457] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:25:08,830 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-64) [12a1573c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:25:08,831 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-64) [12a1573c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 22:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [6651808c] Autorecovering 1 hosts >2013-03-10 22:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [6651808c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [44402540] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [44402540] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:30:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [44402540] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7144854c >2013-03-10 22:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [44402540] FINISH, SetVdsStatusVDSCommand, log id: 7144854c >2013-03-10 22:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [44402540] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7fc282 >2013-03-10 22:30:00,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [44402540] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 724f7e91 >2013-03-10 22:30:00,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [44402540] FINISH, GetHardwareInfoVDSCommand, log id: 724f7e91 >2013-03-10 22:30:00,100 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-96) [6475ff16] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:30:00,103 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-96) [725c4cfc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:30:00,104 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [725c4cfc] FINISH, ActivateVdsVDSCommand, log id: 7fc282 >2013-03-10 22:30:00,124 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [725c4cfc] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:30:00,125 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-96) [725c4cfc] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 22:30:00,126 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [725c4cfc] Autorecovering 1 storage domains >2013-03-10 22:30:00,126 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [725c4cfc] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 22:30:00,127 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [113b057b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:30:00,127 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [113b057b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 10:30 PM >2013-03-10 22:30:00,130 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [113b057b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 10:30 PM >2013-03-10 22:30:00,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [337f8c98] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4340595e >2013-03-10 22:30:00,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [337f8c98] FINISH, GetHardwareInfoVDSCommand, log id: 4340595e >2013-03-10 22:30:00,208 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-49) [18faa439] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 22:30:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-49) [4fe410c9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 22:30:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6930886b >2013-03-10 22:30:06,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 22:30:06,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:30:06,251 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 22:30:06,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 6930886b >2013-03-10 22:30:06,255 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [51620937] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:30:06,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [51620937] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1e35b168 >2013-03-10 22:30:06,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [51620937] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 22:30:06,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [51620937] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1e35b168 >2013-03-10 22:30:06,271 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [51620937] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 22:30:06,271 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [4fe410c9] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 22:30:06,280 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-49) [424a15be] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:30:06,282 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [424a15be] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1cf55290 >2013-03-10 22:30:06,283 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [424a15be] FINISH, SetVdsStatusVDSCommand, log id: 1cf55290 >2013-03-10 22:30:06,289 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-49) [424a15be] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 22:30:06,310 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-49) [2e042066] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:30:06,313 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-49) [6228d5ba] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:30:06,314 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) [6228d5ba] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 22:30:33,857 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-63) [57b553a4] Start refreshing all users data >2013-03-10 22:30:35,013 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-18) [4a1e14e4] Attempting to update VMs/Templates Ovf. >2013-03-10 22:30:35,014 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-18) [4a1e14e4] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 22:30:35,021 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-18) [4a1e14e4] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 22:30:35,021 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-18) [4a1e14e4] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 22:30:35,024 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-18) [4a1e14e4] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 22:30:35,024 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-18) [4a1e14e4] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 22:30:35,025 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-18) [4a1e14e4] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 22:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [1fa5fcca] Autorecovering 1 hosts >2013-03-10 22:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [1fa5fcca] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [15a0a13a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [15a0a13a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [15a0a13a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3830d15b >2013-03-10 22:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [15a0a13a] FINISH, SetVdsStatusVDSCommand, log id: 3830d15b >2013-03-10 22:35:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [15a0a13a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 27274146 >2013-03-10 22:35:00,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [15a0a13a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3de5ca56 >2013-03-10 22:35:00,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [15a0a13a] FINISH, GetHardwareInfoVDSCommand, log id: 3de5ca56 >2013-03-10 22:35:00,262 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-79) [6a2ec455] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:35:00,265 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-79) [2e766e19] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:35:00,266 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [2e766e19] FINISH, ActivateVdsVDSCommand, log id: 27274146 >2013-03-10 22:35:00,285 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [2e766e19] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:35:00,286 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-79) [2e766e19] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 22:35:00,287 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [2e766e19] Autorecovering 1 storage domains >2013-03-10 22:35:00,288 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [2e766e19] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 22:35:00,288 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [35b4f6df] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:35:00,288 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [35b4f6df] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 10:35 PM >2013-03-10 22:35:00,291 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [35b4f6df] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 10:35 PM >2013-03-10 22:35:00,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [41f2564c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 648ff916 >2013-03-10 22:35:00,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [41f2564c] FINISH, GetHardwareInfoVDSCommand, log id: 648ff916 >2013-03-10 22:35:00,622 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-57) [7c6502fd] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 22:35:00,626 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-57) [60b3daf4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 22:35:00,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 40d96004 >2013-03-10 22:35:07,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 22:35:07,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:35:07,619 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 22:35:07,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 40d96004 >2013-03-10 22:35:07,624 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [55fddb89] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:35:07,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [55fddb89] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 19aa9f74 >2013-03-10 22:35:07,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [55fddb89] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 22:35:07,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [55fddb89] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 19aa9f74 >2013-03-10 22:35:07,639 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [55fddb89] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 22:35:07,640 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [60b3daf4] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 22:35:07,649 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-57) [60109afb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:35:07,650 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [60109afb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4461c69 >2013-03-10 22:35:07,652 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [60109afb] FINISH, SetVdsStatusVDSCommand, log id: 4461c69 >2013-03-10 22:35:07,657 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-57) [60109afb] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 22:35:07,680 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-57) [64ac8979] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:35:07,682 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-57) [739bd752] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:35:07,684 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-57) [739bd752] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 22:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [4401fa88] Autorecovering 1 hosts >2013-03-10 22:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [4401fa88] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [322c7041] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [322c7041] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [322c7041] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4bda0a2f >2013-03-10 22:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [322c7041] FINISH, SetVdsStatusVDSCommand, log id: 4bda0a2f >2013-03-10 22:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [322c7041] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7e79a953 >2013-03-10 22:40:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [322c7041] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 549947aa >2013-03-10 22:40:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [322c7041] FINISH, GetHardwareInfoVDSCommand, log id: 549947aa >2013-03-10 22:40:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [256625ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:40:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [f80c05c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:40:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [f80c05c] FINISH, ActivateVdsVDSCommand, log id: 7e79a953 >2013-03-10 22:40:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [f80c05c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:40:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [f80c05c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 22:40:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [f80c05c] Autorecovering 1 storage domains >2013-03-10 22:40:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [f80c05c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 22:40:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [1faccc09] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:40:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [1faccc09] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 10:40 PM >2013-03-10 22:40:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [1faccc09] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 10:40 PM >2013-03-10 22:40:01,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [310fbd06] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5cf9e5fa >2013-03-10 22:40:01,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [310fbd06] FINISH, GetHardwareInfoVDSCommand, log id: 5cf9e5fa >2013-03-10 22:40:01,966 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-31) [26de2904] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 22:40:01,970 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-31) [64224d21] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 22:40:01,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 338db778 >2013-03-10 22:40:07,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 22:40:07,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:40:07,979 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 22:40:07,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 338db778 >2013-03-10 22:40:07,982 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [21586329] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:40:08,003 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [21586329] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 90cb1cc >2013-03-10 22:40:08,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [21586329] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 22:40:08,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [21586329] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 90cb1cc >2013-03-10 22:40:08,006 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [21586329] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 22:40:08,006 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [64224d21] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 22:40:08,015 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-31) [5df4ae2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:40:08,017 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [5df4ae2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2d60f061 >2013-03-10 22:40:08,018 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [5df4ae2] FINISH, SetVdsStatusVDSCommand, log id: 2d60f061 >2013-03-10 22:40:08,024 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-31) [5df4ae2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 22:40:08,045 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-31) [69131ecf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:40:08,048 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-31) [50577c89] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:40:08,049 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-31) [50577c89] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 22:45:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [31966c82] Autorecovering 1 hosts >2013-03-10 22:45:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [31966c82] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:45:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [7054cb58] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:45:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [7054cb58] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [7054cb58] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1a8c7388 >2013-03-10 22:45:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [7054cb58] FINISH, SetVdsStatusVDSCommand, log id: 1a8c7388 >2013-03-10 22:45:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-75) [7054cb58] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 33be5799 >2013-03-10 22:45:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [7054cb58] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2a7215d5 >2013-03-10 22:45:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [7054cb58] FINISH, GetHardwareInfoVDSCommand, log id: 2a7215d5 >2013-03-10 22:45:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [1d34a34d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:45:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [63d035bf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:45:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-75) [63d035bf] FINISH, ActivateVdsVDSCommand, log id: 33be5799 >2013-03-10 22:45:00,227 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [63d035bf] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:45:00,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-75) [63d035bf] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 22:45:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [63d035bf] Autorecovering 1 storage domains >2013-03-10 22:45:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [63d035bf] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 22:45:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [11fce17e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:45:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [11fce17e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 10:45 PM >2013-03-10 22:45:00,237 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [11fce17e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 10:45 PM >2013-03-10 22:45:02,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [3c8afe7d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 52747d64 >2013-03-10 22:45:02,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [3c8afe7d] FINISH, GetHardwareInfoVDSCommand, log id: 52747d64 >2013-03-10 22:45:02,348 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-48) [703f9eb6] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 22:45:02,352 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-48) [1fe8f30a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 22:45:02,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7e39a88d >2013-03-10 22:45:11,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 22:45:11,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:45:11,340 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 22:45:11,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 7e39a88d >2013-03-10 22:45:11,344 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [6c1398d9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:45:11,359 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [6c1398d9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 514f2390 >2013-03-10 22:45:11,359 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [6c1398d9] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 22:45:11,359 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [6c1398d9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 514f2390 >2013-03-10 22:45:11,362 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [6c1398d9] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 22:45:11,362 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [1fe8f30a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 22:45:11,374 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-48) [62f89569] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:45:11,375 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [62f89569] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2966204b >2013-03-10 22:45:11,376 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [62f89569] FINISH, SetVdsStatusVDSCommand, log id: 2966204b >2013-03-10 22:45:11,382 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-48) [62f89569] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 22:45:11,404 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-48) [1d982628] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:45:11,407 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-48) [7b1a4385] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:45:11,408 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-48) [7b1a4385] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 22:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [353d0c29] Autorecovering 1 hosts >2013-03-10 22:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [353d0c29] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [6c2e7c15] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [6c2e7c15] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [6c2e7c15] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 78f1368d >2013-03-10 22:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [6c2e7c15] FINISH, SetVdsStatusVDSCommand, log id: 78f1368d >2013-03-10 22:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-41) [6c2e7c15] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 33979535 >2013-03-10 22:50:00,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [6c2e7c15] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 41976b09 >2013-03-10 22:50:00,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [6c2e7c15] FINISH, GetHardwareInfoVDSCommand, log id: 41976b09 >2013-03-10 22:50:00,208 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-41) [4940cae3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:50:00,210 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-41) [6add7fc3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:50:00,212 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-41) [6add7fc3] FINISH, ActivateVdsVDSCommand, log id: 33979535 >2013-03-10 22:50:00,232 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [6add7fc3] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:50:00,233 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-41) [6add7fc3] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 22:50:00,235 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [6add7fc3] Autorecovering 1 storage domains >2013-03-10 22:50:00,235 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [6add7fc3] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 22:50:00,237 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [58b23b68] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:50:00,237 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [58b23b68] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 10:50 PM >2013-03-10 22:50:00,242 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [58b23b68] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 10:50 PM >2013-03-10 22:50:02,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [35b4f6df] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 26f61f95 >2013-03-10 22:50:02,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [35b4f6df] FINISH, GetHardwareInfoVDSCommand, log id: 26f61f95 >2013-03-10 22:50:02,693 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-79) [5eff698f] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 22:50:02,697 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-79) [471516b9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 22:50:02,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 72fda140 >2013-03-10 22:50:08,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 22:50:08,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:50:08,683 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 22:50:08,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 72fda140 >2013-03-10 22:50:08,687 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [628d901b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:50:08,701 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [628d901b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 634dd068 >2013-03-10 22:50:08,702 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [628d901b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 22:50:08,702 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [628d901b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 634dd068 >2013-03-10 22:50:08,704 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [628d901b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 22:50:08,704 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [471516b9] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 22:50:08,715 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-79) [123d1571] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:50:08,716 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [123d1571] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6bf7afc3 >2013-03-10 22:50:08,717 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [123d1571] FINISH, SetVdsStatusVDSCommand, log id: 6bf7afc3 >2013-03-10 22:50:08,723 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-79) [123d1571] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 22:50:08,746 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-79) [43707df1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:50:08,749 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-79) [6ff7b60c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:50:08,750 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-79) [6ff7b60c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 22:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [54cde22a] Autorecovering 1 hosts >2013-03-10 22:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [54cde22a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [5c397d33] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [5c397d33] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-29) [5c397d33] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7535a8b9 >2013-03-10 22:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-29) [5c397d33] FINISH, SetVdsStatusVDSCommand, log id: 7535a8b9 >2013-03-10 22:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-29) [5c397d33] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4e5c8b2e >2013-03-10 22:55:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-29) [5c397d33] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 390700c1 >2013-03-10 22:55:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-29) [5c397d33] FINISH, GetHardwareInfoVDSCommand, log id: 390700c1 >2013-03-10 22:55:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-29) [55c1ba35] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:55:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-29) [1099ec78] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:55:00,203 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-29) [1099ec78] FINISH, ActivateVdsVDSCommand, log id: 4e5c8b2e >2013-03-10 22:55:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [1099ec78] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 22:55:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-29) [1099ec78] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 22:55:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [1099ec78] Autorecovering 1 storage domains >2013-03-10 22:55:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [1099ec78] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 22:55:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [1b3d25be] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:55:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [1b3d25be] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 10:55 PM >2013-03-10 22:55:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [1b3d25be] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 10:55 PM >2013-03-10 22:55:02,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [d664e39] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 334fa665 >2013-03-10 22:55:02,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [d664e39] FINISH, GetHardwareInfoVDSCommand, log id: 334fa665 >2013-03-10 22:55:03,035 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-25) [6e5f5d66] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 22:55:03,039 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-25) [efeb1ab] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 22:55:03,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 354ef60d >2013-03-10 22:55:09,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 22:55:09,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 22:55:09,090 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 22:55:09,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 354ef60d >2013-03-10 22:55:09,094 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [3a9ba619] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 22:55:09,119 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [3a9ba619] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4e49bb70 >2013-03-10 22:55:09,119 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [3a9ba619] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 22:55:09,119 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [3a9ba619] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4e49bb70 >2013-03-10 22:55:09,121 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [3a9ba619] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 22:55:09,122 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [efeb1ab] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 22:55:09,131 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-25) [1fe02812] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:55:09,132 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [1fe02812] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: b58a3a7 >2013-03-10 22:55:09,134 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [1fe02812] FINISH, SetVdsStatusVDSCommand, log id: b58a3a7 >2013-03-10 22:55:09,141 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-25) [1fe02812] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 22:55:09,180 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-25) [2bbc5c87] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:55:09,183 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-25) [1d64616f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 22:55:09,184 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-25) [1d64616f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 23:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [69fe74ad] Autorecovering 1 hosts >2013-03-10 23:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [69fe74ad] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [6fd24fd9] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [6fd24fd9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [6fd24fd9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 316a9d77 >2013-03-10 23:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-35) [6fd24fd9] FINISH, SetVdsStatusVDSCommand, log id: 316a9d77 >2013-03-10 23:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-35) [6fd24fd9] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 73de4584 >2013-03-10 23:00:00,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [6fd24fd9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 32a306de >2013-03-10 23:00:00,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-35) [6fd24fd9] FINISH, GetHardwareInfoVDSCommand, log id: 32a306de >2013-03-10 23:00:00,208 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-35) [2cd39f13] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:00:00,211 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-35) [58d3d6d1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:00:00,212 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-35) [58d3d6d1] FINISH, ActivateVdsVDSCommand, log id: 73de4584 >2013-03-10 23:00:00,232 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-35) [58d3d6d1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:00:00,233 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-35) [58d3d6d1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 23:00:00,234 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [58d3d6d1] Autorecovering 1 storage domains >2013-03-10 23:00:00,234 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-35) [58d3d6d1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 23:00:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [3c6eb9e5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:00:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [3c6eb9e5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:00 PM >2013-03-10 23:00:00,238 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-35) [3c6eb9e5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:00 PM >2013-03-10 23:00:00,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [192b01be] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4a86ad82 >2013-03-10 23:00:00,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [192b01be] FINISH, GetHardwareInfoVDSCommand, log id: 4a86ad82 >2013-03-10 23:00:00,468 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-83) [10b49dae] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 23:00:00,471 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-83) [e97a44d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 23:00:00,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 732e65ea >2013-03-10 23:00:08,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 23:00:08,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:00:08,517 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 23:00:08,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 732e65ea >2013-03-10 23:00:08,521 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [69abb7fd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:00:08,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [69abb7fd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 62e2a6ce >2013-03-10 23:00:08,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [69abb7fd] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 23:00:08,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [69abb7fd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 62e2a6ce >2013-03-10 23:00:08,537 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [69abb7fd] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 23:00:08,538 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [e97a44d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 23:00:08,556 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-83) [5009b225] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:00:08,557 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [5009b225] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4548e8c2 >2013-03-10 23:00:08,558 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [5009b225] FINISH, SetVdsStatusVDSCommand, log id: 4548e8c2 >2013-03-10 23:00:08,564 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-83) [5009b225] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 23:00:08,586 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-83) [54ac5b64] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:00:08,589 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-83) [b5dcd2e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:00:08,590 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-83) [b5dcd2e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 23:05:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [56f2646] Autorecovering 1 hosts >2013-03-10 23:05:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [56f2646] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-76) [39188db7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-76) [39188db7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [39188db7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: b8a6c1c >2013-03-10 23:05:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [39188db7] FINISH, SetVdsStatusVDSCommand, log id: b8a6c1c >2013-03-10 23:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-76) [39188db7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5d26f507 >2013-03-10 23:05:00,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [39188db7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6a0fe2f5 >2013-03-10 23:05:00,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [39188db7] FINISH, GetHardwareInfoVDSCommand, log id: 6a0fe2f5 >2013-03-10 23:05:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-76) [7d1bd009] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:05:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-76) [60e2c4f6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:05:00,203 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-76) [60e2c4f6] FINISH, ActivateVdsVDSCommand, log id: 5d26f507 >2013-03-10 23:05:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-76) [60e2c4f6] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:05:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [60e2c4f6] Autorecovering 1 storage domains >2013-03-10 23:05:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [60e2c4f6] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 23:05:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-76) [23334eb1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:05:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-76) [23334eb1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:05 PM >2013-03-10 23:05:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-76) [23334eb1] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:05 PM >2013-03-10 23:05:02,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [775e1638] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7289e8e1 >2013-03-10 23:05:02,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [775e1638] FINISH, GetHardwareInfoVDSCommand, log id: 7289e8e1 >2013-03-10 23:05:02,871 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-28) [182088eb] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 23:05:02,875 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-28) [5e75fb02] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 23:05:02,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 15193e0b >2013-03-10 23:05:09,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 23:05:09,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:05:09,882 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 23:05:09,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 15193e0b >2013-03-10 23:05:09,886 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [2e6e7313] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:05:09,900 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [2e6e7313] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7403df70 >2013-03-10 23:05:09,900 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [2e6e7313] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 23:05:09,900 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [2e6e7313] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7403df70 >2013-03-10 23:05:09,902 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [2e6e7313] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 23:05:09,902 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [5e75fb02] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 23:05:09,912 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-28) [7c380f52] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:05:09,913 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [7c380f52] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2052fdb0 >2013-03-10 23:05:09,915 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [7c380f52] FINISH, SetVdsStatusVDSCommand, log id: 2052fdb0 >2013-03-10 23:05:09,921 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-28) [7c380f52] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 23:05:09,945 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-28) [1ddfd829] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:05:09,947 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-28) [442dd2eb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:05:09,948 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-28) [442dd2eb] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 23:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [24449f3] Autorecovering 1 hosts >2013-03-10 23:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [24449f3] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [138e3efd] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [138e3efd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [138e3efd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 46a49086 >2013-03-10 23:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [138e3efd] FINISH, SetVdsStatusVDSCommand, log id: 46a49086 >2013-03-10 23:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-47) [138e3efd] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 656a44e1 >2013-03-10 23:10:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [138e3efd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3e2e8b7e >2013-03-10 23:10:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [138e3efd] FINISH, GetHardwareInfoVDSCommand, log id: 3e2e8b7e >2013-03-10 23:10:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [4b8d8a53] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:10:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [27569bd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:10:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-47) [27569bd] FINISH, ActivateVdsVDSCommand, log id: 656a44e1 >2013-03-10 23:10:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [27569bd] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:10:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [27569bd] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 23:10:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [27569bd] Autorecovering 1 storage domains >2013-03-10 23:10:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [27569bd] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 23:10:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [51132367] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:10:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [51132367] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:10 PM >2013-03-10 23:10:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [51132367] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:10 PM >2013-03-10 23:10:01,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [35f983d2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: b84d645 >2013-03-10 23:10:01,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [35f983d2] FINISH, GetHardwareInfoVDSCommand, log id: b84d645 >2013-03-10 23:10:01,229 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-51) [15e42ffe] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 23:10:01,232 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-51) [23c81f11] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 23:10:01,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1d6b94ed >2013-03-10 23:10:08,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 23:10:08,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:10:08,231 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 23:10:08,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 1d6b94ed >2013-03-10 23:10:08,235 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [6d152c10] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:10:08,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [6d152c10] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 22729283 >2013-03-10 23:10:08,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [6d152c10] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 23:10:08,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [6d152c10] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 22729283 >2013-03-10 23:10:08,251 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [6d152c10] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 23:10:08,252 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [23c81f11] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 23:10:08,261 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-51) [5966fa64] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:10:08,262 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [5966fa64] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5b88354d >2013-03-10 23:10:08,264 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [5966fa64] FINISH, SetVdsStatusVDSCommand, log id: 5b88354d >2013-03-10 23:10:08,269 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-51) [5966fa64] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 23:10:08,292 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-51) [63c228fd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:10:08,294 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-51) [20184aaa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:10:08,296 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-51) [20184aaa] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 23:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [4760c562] Autorecovering 1 hosts >2013-03-10 23:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [4760c562] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [f968f95] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [f968f95] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [f968f95] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5fdd8562 >2013-03-10 23:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [f968f95] FINISH, SetVdsStatusVDSCommand, log id: 5fdd8562 >2013-03-10 23:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-58) [f968f95] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5659b22d >2013-03-10 23:15:00,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [f968f95] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4bf7fab8 >2013-03-10 23:15:00,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [f968f95] FINISH, GetHardwareInfoVDSCommand, log id: 4bf7fab8 >2013-03-10 23:15:00,407 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-58) [7ded9945] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:15:00,410 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-58) [6c89847] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:15:00,411 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-58) [6c89847] FINISH, ActivateVdsVDSCommand, log id: 5659b22d >2013-03-10 23:15:00,430 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [6c89847] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:15:00,431 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-58) [6c89847] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 23:15:00,432 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [6c89847] Autorecovering 1 storage domains >2013-03-10 23:15:00,432 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [6c89847] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 23:15:00,433 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [65aebae5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:15:00,433 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [65aebae5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:15 PM >2013-03-10 23:15:00,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [65aebae5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:15 PM >2013-03-10 23:15:02,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-30) [4e5c16e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3197b0f4 >2013-03-10 23:15:02,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-30) [4e5c16e] FINISH, GetHardwareInfoVDSCommand, log id: 3197b0f4 >2013-03-10 23:15:02,577 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-30) [28202a4a] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 23:15:02,580 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-30) [8a18bb8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 23:15:02,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6bc81958 >2013-03-10 23:15:09,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 23:15:09,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:15:09,580 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 23:15:09,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 6bc81958 >2013-03-10 23:15:09,584 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [3749a967] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:15:09,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [3749a967] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3cb2c065 >2013-03-10 23:15:09,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [3749a967] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 23:15:09,598 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [3749a967] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3cb2c065 >2013-03-10 23:15:09,600 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [3749a967] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 23:15:09,600 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [8a18bb8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 23:15:09,610 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-30) [5cf0f036] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:15:09,612 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-30) [5cf0f036] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 62e5a5be >2013-03-10 23:15:09,613 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-30) [5cf0f036] FINISH, SetVdsStatusVDSCommand, log id: 62e5a5be >2013-03-10 23:15:09,619 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-30) [5cf0f036] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 23:15:09,641 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-30) [45c49a2f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:15:09,643 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-30) [3e9fbe] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:15:09,644 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-30) [3e9fbe] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 23:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [415ffe6a] Autorecovering 1 hosts >2013-03-10 23:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [415ffe6a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [4d039beb] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [4d039beb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [4d039beb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 33428307 >2013-03-10 23:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [4d039beb] FINISH, SetVdsStatusVDSCommand, log id: 33428307 >2013-03-10 23:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [4d039beb] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5e5ad72f >2013-03-10 23:20:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [4d039beb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 429211c6 >2013-03-10 23:20:00,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [4d039beb] FINISH, GetHardwareInfoVDSCommand, log id: 429211c6 >2013-03-10 23:20:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [4c179d55] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:20:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [11b9409c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:20:00,203 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [11b9409c] FINISH, ActivateVdsVDSCommand, log id: 5e5ad72f >2013-03-10 23:20:00,222 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [11b9409c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:20:00,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [11b9409c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 23:20:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [11b9409c] Autorecovering 1 storage domains >2013-03-10 23:20:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [11b9409c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 23:20:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [6dd0e8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:20:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [6dd0e8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:20 PM >2013-03-10 23:20:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [6dd0e8] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:20 PM >2013-03-10 23:20:00,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [4ef5030d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: ef3fa0b >2013-03-10 23:20:00,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [4ef5030d] FINISH, GetHardwareInfoVDSCommand, log id: ef3fa0b >2013-03-10 23:20:00,923 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-65) [2546677f] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 23:20:00,927 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-65) [20a5a439] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 23:20:00,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5e61ea0d >2013-03-10 23:20:07,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 23:20:07,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:20:07,927 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 23:20:07,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 5e61ea0d >2013-03-10 23:20:07,930 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [1cff7e2e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:20:07,944 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [1cff7e2e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4483b690 >2013-03-10 23:20:07,944 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [1cff7e2e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 23:20:07,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [1cff7e2e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4483b690 >2013-03-10 23:20:07,949 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [20a5a439] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 23:20:07,958 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-65) [52bb5c07] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:20:07,960 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [52bb5c07] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 17e8dbe1 >2013-03-10 23:20:07,961 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [52bb5c07] FINISH, SetVdsStatusVDSCommand, log id: 17e8dbe1 >2013-03-10 23:20:07,967 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-65) [52bb5c07] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 23:20:07,988 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-65) [6ce2a26f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:20:07,990 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-65) [3d5118cd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:20:07,992 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-65) [3d5118cd] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 23:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [a9e9ef1] Autorecovering 1 hosts >2013-03-10 23:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [a9e9ef1] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [6b196966] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [6b196966] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [6b196966] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 792b8890 >2013-03-10 23:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [6b196966] FINISH, SetVdsStatusVDSCommand, log id: 792b8890 >2013-03-10 23:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [6b196966] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 40e8dd78 >2013-03-10 23:25:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [6b196966] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 15fddf68 >2013-03-10 23:25:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [6b196966] FINISH, GetHardwareInfoVDSCommand, log id: 15fddf68 >2013-03-10 23:25:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-84) [257602c4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:25:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-84) [551eb4fc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:25:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [551eb4fc] FINISH, ActivateVdsVDSCommand, log id: 40e8dd78 >2013-03-10 23:25:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [551eb4fc] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:25:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-84) [551eb4fc] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 23:25:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [551eb4fc] Autorecovering 1 storage domains >2013-03-10 23:25:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [551eb4fc] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 23:25:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [339fe7f2] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:25:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [339fe7f2] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:25 PM >2013-03-10 23:25:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [339fe7f2] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:25 PM >2013-03-10 23:25:02,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [44b03170] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6da7869a >2013-03-10 23:25:02,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [44b03170] FINISH, GetHardwareInfoVDSCommand, log id: 6da7869a >2013-03-10 23:25:02,264 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-8) [593c8b5f] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 23:25:02,268 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-8) [cf9ef53] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 23:25:02,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3a0d1f7d >2013-03-10 23:25:09,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 23:25:09,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:25:09,265 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 23:25:09,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 3a0d1f7d >2013-03-10 23:25:09,269 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [4e3371de] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:25:09,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4e3371de] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 73c86c19 >2013-03-10 23:25:09,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [4e3371de] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 23:25:09,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4e3371de] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 73c86c19 >2013-03-10 23:25:09,286 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [4e3371de] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 23:25:09,286 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [cf9ef53] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 23:25:09,295 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-8) [428e382] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:25:09,297 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [428e382] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 843e33e >2013-03-10 23:25:09,298 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [428e382] FINISH, SetVdsStatusVDSCommand, log id: 843e33e >2013-03-10 23:25:09,304 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-8) [428e382] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 23:25:09,326 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [3ffe22c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:25:09,329 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [38e98821] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:25:09,330 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-8) [38e98821] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 23:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [5c43f619] Autorecovering 1 hosts >2013-03-10 23:30:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [5c43f619] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [15716762] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [15716762] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [15716762] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4e982ff2 >2013-03-10 23:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [15716762] FINISH, SetVdsStatusVDSCommand, log id: 4e982ff2 >2013-03-10 23:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-93) [15716762] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 62edf4e4 >2013-03-10 23:30:00,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [15716762] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3880bc2c >2013-03-10 23:30:00,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [15716762] FINISH, GetHardwareInfoVDSCommand, log id: 3880bc2c >2013-03-10 23:30:00,410 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [adf0aa1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:30:00,412 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [1ad3b720] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:30:00,414 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-93) [1ad3b720] FINISH, ActivateVdsVDSCommand, log id: 62edf4e4 >2013-03-10 23:30:00,433 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [1ad3b720] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:30:00,434 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-93) [1ad3b720] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 23:30:00,435 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [1ad3b720] Autorecovering 1 storage domains >2013-03-10 23:30:00,435 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [1ad3b720] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 23:30:00,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [1849eaac] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:30:00,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [1849eaac] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:30 PM >2013-03-10 23:30:00,439 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [1849eaac] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:30 PM >2013-03-10 23:30:00,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [1212d4d0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 65a464b2 >2013-03-10 23:30:00,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [1212d4d0] FINISH, GetHardwareInfoVDSCommand, log id: 65a464b2 >2013-03-10 23:30:00,612 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-12) [4bb98b1a] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 23:30:00,615 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-12) [1b0c670b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 23:30:00,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 15ba5939 >2013-03-10 23:30:07,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 23:30:07,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:30:07,644 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 23:30:07,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 15ba5939 >2013-03-10 23:30:07,647 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [73d00ecc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:30:07,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [73d00ecc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6d8b3cb2 >2013-03-10 23:30:07,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [73d00ecc] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 23:30:07,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [73d00ecc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6d8b3cb2 >2013-03-10 23:30:07,677 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [73d00ecc] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 23:30:07,677 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [1b0c670b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 23:30:07,697 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-12) [36389571] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:30:07,699 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [36389571] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 213d2b45 >2013-03-10 23:30:07,700 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [36389571] FINISH, SetVdsStatusVDSCommand, log id: 213d2b45 >2013-03-10 23:30:07,705 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-12) [36389571] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 23:30:07,745 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-12) [1414911f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:30:07,748 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-12) [5a57de2b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:30:07,749 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-12) [5a57de2b] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 23:30:33,860 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-61) [53ecd05b] Start refreshing all users data >2013-03-10 23:30:35,026 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-26) [1faccc09] Attempting to update VMs/Templates Ovf. >2013-03-10 23:30:35,027 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-26) [1faccc09] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 23:30:35,034 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-26) [1faccc09] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 23:30:35,034 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-26) [1faccc09] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 23:30:35,037 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-26) [1faccc09] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 23:30:35,037 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-26) [1faccc09] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 23:30:35,038 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-26) [1faccc09] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-10 23:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [104ef06f] Autorecovering 1 hosts >2013-03-10 23:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [104ef06f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [63f2b9d7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [63f2b9d7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [63f2b9d7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 18bfe69c >2013-03-10 23:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [63f2b9d7] FINISH, SetVdsStatusVDSCommand, log id: 18bfe69c >2013-03-10 23:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [63f2b9d7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 23cc9a01 >2013-03-10 23:35:00,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [63f2b9d7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3ef264a0 >2013-03-10 23:35:00,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [63f2b9d7] FINISH, GetHardwareInfoVDSCommand, log id: 3ef264a0 >2013-03-10 23:35:00,232 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [280800bf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:35:00,234 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [7d889c77] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:35:00,235 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [7d889c77] FINISH, ActivateVdsVDSCommand, log id: 23cc9a01 >2013-03-10 23:35:00,255 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [7d889c77] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:35:00,256 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [7d889c77] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 23:35:00,257 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [7d889c77] Autorecovering 1 storage domains >2013-03-10 23:35:00,257 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [7d889c77] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 23:35:00,258 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [41b549dc] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:35:00,258 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [41b549dc] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:35 PM >2013-03-10 23:35:00,260 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [41b549dc] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:35 PM >2013-03-10 23:35:01,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [21b2cbb7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 54d1afaa >2013-03-10 23:35:01,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [21b2cbb7] FINISH, GetHardwareInfoVDSCommand, log id: 54d1afaa >2013-03-10 23:35:02,034 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-22) [42f66459] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 23:35:02,038 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-22) [1291fa5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 23:35:02,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 30cf396 >2013-03-10 23:35:07,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 23:35:07,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:35:07,024 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 23:35:07,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 30cf396 >2013-03-10 23:35:07,028 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [26863ec5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:35:07,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [26863ec5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2aeba666 >2013-03-10 23:35:07,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [26863ec5] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 23:35:07,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [26863ec5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2aeba666 >2013-03-10 23:35:07,044 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [26863ec5] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 23:35:07,044 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [1291fa5] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 23:35:07,053 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-22) [10853955] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:35:07,055 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [10853955] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2649f183 >2013-03-10 23:35:07,056 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [10853955] FINISH, SetVdsStatusVDSCommand, log id: 2649f183 >2013-03-10 23:35:07,062 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [10853955] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 23:35:07,084 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [554a9dda] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:35:07,087 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [525b6a6c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:35:07,090 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-22) [525b6a6c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 23:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [339fe7f2] Autorecovering 1 hosts >2013-03-10 23:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [339fe7f2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [40326a1e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [40326a1e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:40:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [40326a1e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 39bbaf0 >2013-03-10 23:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [40326a1e] FINISH, SetVdsStatusVDSCommand, log id: 39bbaf0 >2013-03-10 23:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [40326a1e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7973b561 >2013-03-10 23:40:00,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [40326a1e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2a77464f >2013-03-10 23:40:00,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [40326a1e] FINISH, GetHardwareInfoVDSCommand, log id: 2a77464f >2013-03-10 23:40:00,563 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-84) [26e6ee0d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:40:00,566 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-84) [3350fdc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:40:00,567 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [3350fdc] FINISH, ActivateVdsVDSCommand, log id: 7973b561 >2013-03-10 23:40:00,586 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [3350fdc] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:40:00,587 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-84) [3350fdc] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 23:40:00,588 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [3350fdc] Autorecovering 1 storage domains >2013-03-10 23:40:00,588 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [3350fdc] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 23:40:00,589 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [7be57232] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:40:00,589 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [7be57232] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:40 PM >2013-03-10 23:40:00,592 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [7be57232] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:40 PM >2013-03-10 23:40:01,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [51132367] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3a0ae7ce >2013-03-10 23:40:01,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [51132367] FINISH, GetHardwareInfoVDSCommand, log id: 3a0ae7ce >2013-03-10 23:40:01,373 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-47) [6864976c] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 23:40:01,376 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-47) [4825936] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 23:40:01,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 69c8c442 >2013-03-10 23:40:07,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 23:40:07,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:40:07,387 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 23:40:07,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 69c8c442 >2013-03-10 23:40:07,391 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [4e403e42] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:40:07,404 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4e403e42] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 23ab1f0f >2013-03-10 23:40:07,404 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [4e403e42] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 23:40:07,404 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4e403e42] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 23ab1f0f >2013-03-10 23:40:07,406 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [4e403e42] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 23:40:07,407 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [4825936] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 23:40:07,416 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-47) [70997e2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:40:07,418 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [70997e2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 330bac1 >2013-03-10 23:40:07,419 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [70997e2] FINISH, SetVdsStatusVDSCommand, log id: 330bac1 >2013-03-10 23:40:07,425 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [70997e2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 23:40:07,447 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [17da42e1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:40:07,450 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [7fb66044] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:40:07,451 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-47) [7fb66044] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 23:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-40) [7a20673a] Autorecovering 1 hosts >2013-03-10 23:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-40) [7a20673a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-40) [66aabdbf] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-40) [66aabdbf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-40) [66aabdbf] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7b28d3c2 >2013-03-10 23:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-40) [66aabdbf] FINISH, SetVdsStatusVDSCommand, log id: 7b28d3c2 >2013-03-10 23:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-40) [66aabdbf] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7683c39 >2013-03-10 23:45:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-40) [66aabdbf] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 77c7db22 >2013-03-10 23:45:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-40) [66aabdbf] FINISH, GetHardwareInfoVDSCommand, log id: 77c7db22 >2013-03-10 23:45:00,087 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-40) [70987a0d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:45:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-40) [5b8ff951] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:45:00,091 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-40) [5b8ff951] FINISH, ActivateVdsVDSCommand, log id: 7683c39 >2013-03-10 23:45:00,110 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-40) [5b8ff951] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:45:00,111 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-40) [5b8ff951] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 23:45:00,112 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-40) [5b8ff951] Autorecovering 1 storage domains >2013-03-10 23:45:00,112 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-40) [5b8ff951] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 23:45:00,113 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-40) [f452ece] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:45:00,113 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-40) [f452ece] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:45 PM >2013-03-10 23:45:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-40) [f452ece] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:45 PM >2013-03-10 23:45:01,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [65aebae5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5a302724 >2013-03-10 23:45:01,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [65aebae5] FINISH, GetHardwareInfoVDSCommand, log id: 5a302724 >2013-03-10 23:45:01,739 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-58) [2d5560bf] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 23:45:01,742 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-58) [1b8c2a30] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 23:45:01,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6e953938 >2013-03-10 23:45:08,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 23:45:08,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:45:08,737 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 23:45:08,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 6e953938 >2013-03-10 23:45:08,741 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [362061b3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:45:08,755 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [362061b3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1a7990ef >2013-03-10 23:45:08,755 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [362061b3] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 23:45:08,755 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [362061b3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1a7990ef >2013-03-10 23:45:08,757 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [362061b3] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 23:45:08,758 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [1b8c2a30] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 23:45:08,768 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-58) [20439f47] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:45:08,769 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [20439f47] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3130ebde >2013-03-10 23:45:08,771 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [20439f47] FINISH, SetVdsStatusVDSCommand, log id: 3130ebde >2013-03-10 23:45:08,777 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-58) [20439f47] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 23:45:08,804 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-58) [3438744] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:45:08,807 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-58) [8f8b439] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:45:08,808 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-58) [8f8b439] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 23:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [361df729] Autorecovering 1 hosts >2013-03-10 23:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [361df729] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:50:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [4442433c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:50:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [4442433c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:50:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [4442433c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: a2cce4a >2013-03-10 23:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [4442433c] FINISH, SetVdsStatusVDSCommand, log id: a2cce4a >2013-03-10 23:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [4442433c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 764e0a8 >2013-03-10 23:50:02,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [4442433c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 18b10b65 >2013-03-10 23:50:02,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [4442433c] FINISH, GetHardwareInfoVDSCommand, log id: 18b10b65 >2013-03-10 23:50:02,410 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [11715ad] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:50:02,413 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [662c02c4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:50:02,414 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [662c02c4] FINISH, ActivateVdsVDSCommand, log id: 764e0a8 >2013-03-10 23:50:02,433 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [662c02c4] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:50:02,434 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [662c02c4] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 23:50:02,435 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [662c02c4] Autorecovering 1 storage domains >2013-03-10 23:50:02,435 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [662c02c4] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 23:50:02,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [5c264d10] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:50:02,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [5c264d10] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:50 PM >2013-03-10 23:50:02,439 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [5c264d10] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:50 PM >2013-03-10 23:50:03,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [6dd0e8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 67e63d2d >2013-03-10 23:50:03,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [6dd0e8] FINISH, GetHardwareInfoVDSCommand, log id: 67e63d2d >2013-03-10 23:50:03,084 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-59) [7cee1c90] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 23:50:03,087 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-59) [3c0133b0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 23:50:03,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7cf1f22c >2013-03-10 23:50:10,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 23:50:10,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:50:10,070 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 23:50:10,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 7cf1f22c >2013-03-10 23:50:10,074 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [14db1950] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:50:10,088 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [14db1950] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3193ed3d >2013-03-10 23:50:10,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [14db1950] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 23:50:10,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [14db1950] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3193ed3d >2013-03-10 23:50:10,091 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [14db1950] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 23:50:10,091 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [3c0133b0] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 23:50:10,100 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-59) [131ad16c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:50:10,102 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [131ad16c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 78236a63 >2013-03-10 23:50:10,103 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [131ad16c] FINISH, SetVdsStatusVDSCommand, log id: 78236a63 >2013-03-10 23:50:10,108 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [131ad16c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 23:50:10,131 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [89ea2d6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:50:10,134 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [64e9fe5f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:50:10,135 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-59) [64e9fe5f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-10 23:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [77f9bcda] Autorecovering 1 hosts >2013-03-10 23:55:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [77f9bcda] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [245e17c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [245e17c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [245e17c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6322cfc3 >2013-03-10 23:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [245e17c] FINISH, SetVdsStatusVDSCommand, log id: 6322cfc3 >2013-03-10 23:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [245e17c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 15f4ad66 >2013-03-10 23:55:00,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [245e17c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6be3c67e >2013-03-10 23:55:00,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [245e17c] FINISH, GetHardwareInfoVDSCommand, log id: 6be3c67e >2013-03-10 23:55:00,100 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-45) [2d709bee] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:55:00,102 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-45) [b632c9a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:55:00,104 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [b632c9a] FINISH, ActivateVdsVDSCommand, log id: 15f4ad66 >2013-03-10 23:55:00,123 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [b632c9a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-10 23:55:00,124 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-45) [b632c9a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-10 23:55:00,125 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [b632c9a] Autorecovering 1 storage domains >2013-03-10 23:55:00,125 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [b632c9a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-10 23:55:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [70173f2e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:55:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [70173f2e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/10/13 11:55 PM >2013-03-10 23:55:00,129 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [70173f2e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/10/13 11:55 PM >2013-03-10 23:55:01,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [7cac8d58] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 10dd668b >2013-03-10 23:55:01,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [7cac8d58] FINISH, GetHardwareInfoVDSCommand, log id: 10dd668b >2013-03-10 23:55:01,405 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-27) [55b2f6be] Running command: InitVdsOnUpCommand internal: true. >2013-03-10 23:55:01,409 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-27) [5118a690] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-10 23:55:01,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5dbe5b9 >2013-03-10 23:55:06,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-10 23:55:06,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-10 23:55:06,468 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-10 23:55:06,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 5dbe5b9 >2013-03-10 23:55:06,472 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [53741b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-10 23:55:06,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [53741b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c86bb76 >2013-03-10 23:55:06,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [53741b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-10 23:55:06,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [53741b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c86bb76 >2013-03-10 23:55:06,488 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [53741b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-10 23:55:06,488 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [5118a690] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-10 23:55:06,498 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-27) [6a3e99f3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:55:06,499 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [6a3e99f3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 22ed5d19 >2013-03-10 23:55:06,501 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [6a3e99f3] FINISH, SetVdsStatusVDSCommand, log id: 22ed5d19 >2013-03-10 23:55:06,506 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-27) [6a3e99f3] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-10 23:55:06,533 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-27) [1b29d46a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:55:06,535 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-27) [6a82214e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-10 23:55:06,537 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-27) [6a82214e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 00:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [49ff21ac] Autorecovering 1 hosts >2013-03-11 00:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [49ff21ac] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-82) [62ef96af] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-82) [62ef96af] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:00:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [62ef96af] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2708a5da >2013-03-11 00:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [62ef96af] FINISH, SetVdsStatusVDSCommand, log id: 2708a5da >2013-03-11 00:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-82) [62ef96af] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2146f0fb >2013-03-11 00:00:00,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [62ef96af] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6782bd3c >2013-03-11 00:00:00,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [62ef96af] FINISH, GetHardwareInfoVDSCommand, log id: 6782bd3c >2013-03-11 00:00:00,096 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-82) [64a14788] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:00:00,098 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-82) [244ecfeb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:00:00,099 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-82) [244ecfeb] FINISH, ActivateVdsVDSCommand, log id: 2146f0fb >2013-03-11 00:00:00,119 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-82) [244ecfeb] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:00:00,120 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-82) [244ecfeb] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 00:00:00,121 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [244ecfeb] Autorecovering 1 storage domains >2013-03-11 00:00:00,121 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [244ecfeb] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 00:00:00,122 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-82) [2e993d7d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:00:00,122 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-82) [2e993d7d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:00 AM >2013-03-11 00:00:00,124 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-82) [2e993d7d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:00 AM >2013-03-11 00:00:00,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [64cdd242] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 197b278c >2013-03-11 00:00:00,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [64cdd242] FINISH, GetHardwareInfoVDSCommand, log id: 197b278c >2013-03-11 00:00:00,811 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-60) [43d7be16] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 00:00:00,814 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-60) [cb6a400] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 00:00:00,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 415becd1 >2013-03-11 00:00:08,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 00:00:08,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:00:08,869 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 00:00:08,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 415becd1 >2013-03-11 00:00:08,873 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [125b6fee] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:00:08,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [125b6fee] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2f06faa7 >2013-03-11 00:00:08,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [125b6fee] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 00:00:08,894 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [125b6fee] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2f06faa7 >2013-03-11 00:00:08,896 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [125b6fee] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 00:00:08,896 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [cb6a400] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 00:00:08,906 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-60) [106d7201] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:00:08,907 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [106d7201] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 481255e7 >2013-03-11 00:00:08,908 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [106d7201] FINISH, SetVdsStatusVDSCommand, log id: 481255e7 >2013-03-11 00:00:08,914 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-60) [106d7201] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 00:00:08,945 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-60) [3da91e03] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:00:08,948 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-60) [616a5905] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:00:08,949 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-60) [616a5905] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 00:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [4da39129] Autorecovering 1 hosts >2013-03-11 00:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [4da39129] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [664552fb] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [664552fb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [664552fb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5a121da7 >2013-03-11 00:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [664552fb] FINISH, SetVdsStatusVDSCommand, log id: 5a121da7 >2013-03-11 00:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-13) [664552fb] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2d5fdc0 >2013-03-11 00:05:00,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [664552fb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 59c7b108 >2013-03-11 00:05:00,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [664552fb] FINISH, GetHardwareInfoVDSCommand, log id: 59c7b108 >2013-03-11 00:05:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-13) [7ab4bf1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:05:00,095 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-13) [54dfcc3e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:05:00,096 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-13) [54dfcc3e] FINISH, ActivateVdsVDSCommand, log id: 2d5fdc0 >2013-03-11 00:05:00,116 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [54dfcc3e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:05:00,117 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-13) [54dfcc3e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 00:05:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [54dfcc3e] Autorecovering 1 storage domains >2013-03-11 00:05:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [54dfcc3e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 00:05:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [2cbdecd0] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:05:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [2cbdecd0] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:05 AM >2013-03-11 00:05:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [2cbdecd0] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:05 AM >2013-03-11 00:05:00,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [681b38ac] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 508e475c >2013-03-11 00:05:00,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [681b38ac] FINISH, GetHardwareInfoVDSCommand, log id: 508e475c >2013-03-11 00:05:00,227 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-33) [21d8734] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 00:05:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-33) [1d593f21] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 00:05:00,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3d2d362d >2013-03-11 00:05:07,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 00:05:07,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:05:07,583 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 00:05:07,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 3d2d362d >2013-03-11 00:05:07,586 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [71c73d90] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:05:07,600 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [71c73d90] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 73681c67 >2013-03-11 00:05:07,600 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [71c73d90] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 00:05:07,600 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [71c73d90] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 73681c67 >2013-03-11 00:05:07,602 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [71c73d90] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 00:05:07,602 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [1d593f21] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 00:05:07,612 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-33) [50968703] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:05:07,613 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [50968703] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5d1ec0b1 >2013-03-11 00:05:07,615 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [50968703] FINISH, SetVdsStatusVDSCommand, log id: 5d1ec0b1 >2013-03-11 00:05:07,621 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-33) [50968703] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 00:05:07,644 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-33) [61ef0ca5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:05:07,647 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-33) [8051887] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:05:07,648 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-33) [8051887] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 00:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [652270c0] Autorecovering 1 hosts >2013-03-11 00:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [652270c0] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [702710dc] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [702710dc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [702710dc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 22534b80 >2013-03-11 00:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [702710dc] FINISH, SetVdsStatusVDSCommand, log id: 22534b80 >2013-03-11 00:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-68) [702710dc] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7201123e >2013-03-11 00:10:00,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [702710dc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 289e22d1 >2013-03-11 00:10:00,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [702710dc] FINISH, GetHardwareInfoVDSCommand, log id: 289e22d1 >2013-03-11 00:10:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-68) [c9c5e79] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:10:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-68) [703d43e3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:10:00,094 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-68) [703d43e3] FINISH, ActivateVdsVDSCommand, log id: 7201123e >2013-03-11 00:10:00,113 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-68) [703d43e3] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:10:00,114 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-68) [703d43e3] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 00:10:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [703d43e3] Autorecovering 1 storage domains >2013-03-11 00:10:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-68) [703d43e3] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 00:10:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [39fe4430] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:10:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [39fe4430] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:10 AM >2013-03-11 00:10:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-68) [39fe4430] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:10 AM >2013-03-11 00:10:01,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [1435b523] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: c00c6f1 >2013-03-11 00:10:01,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [1435b523] FINISH, GetHardwareInfoVDSCommand, log id: c00c6f1 >2013-03-11 00:10:01,928 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-97) [cc15ba7] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 00:10:01,934 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-97) [7085165b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 00:10:01,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3c2f698f >2013-03-11 00:10:09,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 00:10:09,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:10:09,959 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 00:10:09,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 3c2f698f >2013-03-11 00:10:09,963 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [53fa2bbc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:10:09,976 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [53fa2bbc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 296c566b >2013-03-11 00:10:09,976 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [53fa2bbc] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 00:10:09,976 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [53fa2bbc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 296c566b >2013-03-11 00:10:09,978 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [53fa2bbc] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 00:10:09,979 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [7085165b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 00:10:09,989 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-97) [e3d134f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:10:09,990 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [e3d134f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1acebaff >2013-03-11 00:10:09,992 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [e3d134f] FINISH, SetVdsStatusVDSCommand, log id: 1acebaff >2013-03-11 00:10:09,997 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-97) [e3d134f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 00:10:10,020 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-97) [3cac8578] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:10:10,023 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-97) [4dc160a6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:10:10,024 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-97) [4dc160a6] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 00:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [413384bd] Autorecovering 1 hosts >2013-03-11 00:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [413384bd] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [3da7fb9d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [3da7fb9d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [3da7fb9d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 336b6429 >2013-03-11 00:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [3da7fb9d] FINISH, SetVdsStatusVDSCommand, log id: 336b6429 >2013-03-11 00:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [3da7fb9d] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 605f3cff >2013-03-11 00:15:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [3da7fb9d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 17d96a70 >2013-03-11 00:15:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [3da7fb9d] FINISH, GetHardwareInfoVDSCommand, log id: 17d96a70 >2013-03-11 00:15:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [1b2877e1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:15:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [2a8a4fb1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:15:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [2a8a4fb1] FINISH, ActivateVdsVDSCommand, log id: 605f3cff >2013-03-11 00:15:00,112 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [2a8a4fb1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:15:00,113 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [2a8a4fb1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 00:15:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [2a8a4fb1] Autorecovering 1 storage domains >2013-03-11 00:15:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [2a8a4fb1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 00:15:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [111d3efa] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:15:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [111d3efa] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:15 AM >2013-03-11 00:15:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [111d3efa] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:15 AM >2013-03-11 00:15:01,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [3a3123f8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: e0dbbbe >2013-03-11 00:15:01,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [3a3123f8] FINISH, GetHardwareInfoVDSCommand, log id: e0dbbbe >2013-03-11 00:15:01,304 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-11) [4ec52887] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 00:15:01,308 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-11) [326d5e80] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 00:15:01,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 34744a7a >2013-03-11 00:15:05,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 00:15:05,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:15:05,299 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 00:15:05,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 34744a7a >2013-03-11 00:15:05,303 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [96b2b28] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:15:05,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [96b2b28] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 186c57af >2013-03-11 00:15:05,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [96b2b28] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 00:15:05,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [96b2b28] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 186c57af >2013-03-11 00:15:05,319 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [96b2b28] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 00:15:05,320 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [326d5e80] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 00:15:05,329 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-11) [1df5c87f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:15:05,330 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [1df5c87f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 32d3fd8f >2013-03-11 00:15:05,331 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [1df5c87f] FINISH, SetVdsStatusVDSCommand, log id: 32d3fd8f >2013-03-11 00:15:05,337 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-11) [1df5c87f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 00:15:05,359 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-11) [7f870392] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:15:05,362 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-11) [35ad7cab] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:15:05,363 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-11) [35ad7cab] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 00:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [2b6a4589] Autorecovering 1 hosts >2013-03-11 00:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [2b6a4589] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [12d40e9a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [12d40e9a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [12d40e9a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 596ae69 >2013-03-11 00:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [12d40e9a] FINISH, SetVdsStatusVDSCommand, log id: 596ae69 >2013-03-11 00:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-87) [12d40e9a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 144dc327 >2013-03-11 00:20:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [12d40e9a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 133b3102 >2013-03-11 00:20:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [12d40e9a] FINISH, GetHardwareInfoVDSCommand, log id: 133b3102 >2013-03-11 00:20:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [fa4832e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:20:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [4cee74a7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:20:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-87) [4cee74a7] FINISH, ActivateVdsVDSCommand, log id: 144dc327 >2013-03-11 00:20:00,111 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-87) [4cee74a7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:20:00,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [4cee74a7] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 00:20:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [4cee74a7] Autorecovering 1 storage domains >2013-03-11 00:20:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-87) [4cee74a7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 00:20:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [7b36458a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:20:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [7b36458a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:20 AM >2013-03-11 00:20:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-87) [7b36458a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:20 AM >2013-03-11 00:20:02,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [7750554c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 65f108a4 >2013-03-11 00:20:02,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [7750554c] FINISH, GetHardwareInfoVDSCommand, log id: 65f108a4 >2013-03-11 00:20:02,647 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-15) [d9e5e0d] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 00:20:02,650 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-15) [51591cd0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 00:20:02,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3d6c1493 >2013-03-11 00:20:09,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 00:20:09,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:20:09,636 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 00:20:09,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 3d6c1493 >2013-03-11 00:20:09,639 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [7fbecafb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:20:09,653 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [7fbecafb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 637c2c1e >2013-03-11 00:20:09,653 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [7fbecafb] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 00:20:09,653 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [7fbecafb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 637c2c1e >2013-03-11 00:20:09,663 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [51591cd0] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 00:20:09,672 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-15) [7c9d6158] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:20:09,674 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [7c9d6158] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 24daea62 >2013-03-11 00:20:09,675 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [7c9d6158] FINISH, SetVdsStatusVDSCommand, log id: 24daea62 >2013-03-11 00:20:09,681 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-15) [7c9d6158] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 00:20:09,703 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-15) [dfa817a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:20:09,705 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-15) [5664a7d8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:20:09,706 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-15) [5664a7d8] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 00:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [6f06a147] Autorecovering 1 hosts >2013-03-11 00:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [6f06a147] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-86) [7330db01] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-86) [7330db01] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [7330db01] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3b2884c2 >2013-03-11 00:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [7330db01] FINISH, SetVdsStatusVDSCommand, log id: 3b2884c2 >2013-03-11 00:25:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-86) [7330db01] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 51728dbc >2013-03-11 00:25:00,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [7330db01] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 34b994da >2013-03-11 00:25:00,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [7330db01] FINISH, GetHardwareInfoVDSCommand, log id: 34b994da >2013-03-11 00:25:00,269 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-86) [6c00f7ea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:25:00,272 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-86) [271fb1dc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:25:00,273 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-86) [271fb1dc] FINISH, ActivateVdsVDSCommand, log id: 51728dbc >2013-03-11 00:25:00,293 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-86) [271fb1dc] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:25:00,294 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-86) [271fb1dc] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 00:25:00,295 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [271fb1dc] Autorecovering 1 storage domains >2013-03-11 00:25:00,295 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [271fb1dc] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 00:25:00,295 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-86) [36102d4e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:25:00,296 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-86) [36102d4e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:25 AM >2013-03-11 00:25:00,298 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-86) [36102d4e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:25 AM >2013-03-11 00:25:00,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [668aa132] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1608a7a4 >2013-03-11 00:25:00,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [668aa132] FINISH, GetHardwareInfoVDSCommand, log id: 1608a7a4 >2013-03-11 00:25:00,977 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-1) [1d0886e2] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 00:25:00,980 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-1) [35cc5947] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 00:25:00,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: ffd456c >2013-03-11 00:25:05,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 00:25:05,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:25:05,964 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 00:25:05,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: ffd456c >2013-03-11 00:25:05,968 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [f354360] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:25:05,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [f354360] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 14875265 >2013-03-11 00:25:05,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [f354360] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 00:25:05,982 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [f354360] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 14875265 >2013-03-11 00:25:05,984 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [f354360] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 00:25:05,984 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [35cc5947] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 00:25:05,993 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-1) [38fc4bbb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:25:05,995 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [38fc4bbb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 602c36e4 >2013-03-11 00:25:05,996 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [38fc4bbb] FINISH, SetVdsStatusVDSCommand, log id: 602c36e4 >2013-03-11 00:25:06,002 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-1) [38fc4bbb] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 00:25:06,024 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-1) [7d079c5e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:25:06,026 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-1) [6335001a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:25:06,028 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-1) [6335001a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 00:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [40525bec] Autorecovering 1 hosts >2013-03-11 00:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [40525bec] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [67db1bb2] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [67db1bb2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [67db1bb2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 12c9e1c6 >2013-03-11 00:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [67db1bb2] FINISH, SetVdsStatusVDSCommand, log id: 12c9e1c6 >2013-03-11 00:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-90) [67db1bb2] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3d00f388 >2013-03-11 00:30:00,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [67db1bb2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4b1d4fce >2013-03-11 00:30:00,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [67db1bb2] FINISH, GetHardwareInfoVDSCommand, log id: 4b1d4fce >2013-03-11 00:30:00,217 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-90) [3188efa0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:30:00,220 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-90) [5d2ed82] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:30:00,221 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-90) [5d2ed82] FINISH, ActivateVdsVDSCommand, log id: 3d00f388 >2013-03-11 00:30:00,241 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [5d2ed82] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:30:00,242 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [5d2ed82] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 00:30:00,243 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [5d2ed82] Autorecovering 1 storage domains >2013-03-11 00:30:00,243 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [5d2ed82] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 00:30:00,244 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [1e5defb2] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:30:00,244 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [1e5defb2] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:30 AM >2013-03-11 00:30:00,247 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [1e5defb2] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:30 AM >2013-03-11 00:30:00,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [4d280997] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 117a7038 >2013-03-11 00:30:00,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [4d280997] FINISH, GetHardwareInfoVDSCommand, log id: 117a7038 >2013-03-11 00:30:00,319 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-2) [55b3d9b2] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 00:30:00,322 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-2) [48ca881c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 00:30:00,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5a553cd3 >2013-03-11 00:30:07,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 00:30:07,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:30:07,364 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 00:30:07,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 5a553cd3 >2013-03-11 00:30:07,368 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [5a55ebc6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:30:07,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [5a55ebc6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 426b4845 >2013-03-11 00:30:07,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [5a55ebc6] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 00:30:07,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [5a55ebc6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 426b4845 >2013-03-11 00:30:07,384 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [5a55ebc6] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 00:30:07,385 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [48ca881c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 00:30:07,394 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-2) [9b78278] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:30:07,396 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [9b78278] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 27aab67c >2013-03-11 00:30:07,397 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [9b78278] FINISH, SetVdsStatusVDSCommand, log id: 27aab67c >2013-03-11 00:30:07,402 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [9b78278] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 00:30:07,425 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [7ff5a7fe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:30:07,428 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [866e613] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:30:07,429 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-2) [866e613] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 00:30:33,862 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-45) [70173f2e] Start refreshing all users data >2013-03-11 00:30:35,038 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) [7be57232] Attempting to update VMs/Templates Ovf. >2013-03-11 00:30:35,039 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) [7be57232] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 00:30:35,040 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) [7be57232] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 00:30:35,040 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) [7be57232] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 00:30:35,041 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) [7be57232] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 00:30:35,041 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) [7be57232] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 00:30:35,041 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-84) [7be57232] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 00:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [48b870e1] Autorecovering 1 hosts >2013-03-11 00:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [48b870e1] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [2723aefa] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [2723aefa] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [2723aefa] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1ad1ba1c >2013-03-11 00:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [2723aefa] FINISH, SetVdsStatusVDSCommand, log id: 1ad1ba1c >2013-03-11 00:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-88) [2723aefa] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 69c768a7 >2013-03-11 00:35:00,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [2723aefa] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 65cf663d >2013-03-11 00:35:00,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [2723aefa] FINISH, GetHardwareInfoVDSCommand, log id: 65cf663d >2013-03-11 00:35:00,356 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-88) [3501b937] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:35:00,359 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-88) [646411da] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:35:00,360 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-88) [646411da] FINISH, ActivateVdsVDSCommand, log id: 69c768a7 >2013-03-11 00:35:00,380 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [646411da] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:35:00,380 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-88) [646411da] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 00:35:00,382 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [646411da] Autorecovering 1 storage domains >2013-03-11 00:35:00,382 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [646411da] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 00:35:00,382 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [f44a40d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:35:00,382 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [f44a40d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:35 AM >2013-03-11 00:35:00,385 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [f44a40d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:35 AM >2013-03-11 00:35:01,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [2baffae0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 37e2e14 >2013-03-11 00:35:01,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [2baffae0] FINISH, GetHardwareInfoVDSCommand, log id: 37e2e14 >2013-03-11 00:35:01,712 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-78) [62adeb7d] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 00:35:01,715 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-78) [6f21313d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 00:35:01,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2215c423 >2013-03-11 00:35:07,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 00:35:07,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:35:07,721 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 00:35:07,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 2215c423 >2013-03-11 00:35:07,725 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [29e00967] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:35:07,739 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [29e00967] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3eeebbb9 >2013-03-11 00:35:07,739 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [29e00967] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 00:35:07,739 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [29e00967] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3eeebbb9 >2013-03-11 00:35:07,741 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [29e00967] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 00:35:07,741 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [6f21313d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 00:35:07,758 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-78) [1e05ab18] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:35:07,760 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [1e05ab18] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2721ea7 >2013-03-11 00:35:07,761 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [1e05ab18] FINISH, SetVdsStatusVDSCommand, log id: 2721ea7 >2013-03-11 00:35:07,766 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-78) [1e05ab18] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 00:35:07,789 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-78) [260d8381] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:35:07,791 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-78) [1a7544b0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:35:07,792 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-78) [1a7544b0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 00:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [36102d4e] Autorecovering 1 hosts >2013-03-11 00:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [36102d4e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-86) [24e744ac] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-86) [24e744ac] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:40:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [24e744ac] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 23ff4656 >2013-03-11 00:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [24e744ac] FINISH, SetVdsStatusVDSCommand, log id: 23ff4656 >2013-03-11 00:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-86) [24e744ac] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3bd8510d >2013-03-11 00:40:00,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [24e744ac] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 56dc2429 >2013-03-11 00:40:00,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [24e744ac] FINISH, GetHardwareInfoVDSCommand, log id: 56dc2429 >2013-03-11 00:40:00,279 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-86) [52286ffb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:40:00,282 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-86) [1c9de79] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:40:00,283 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-86) [1c9de79] FINISH, ActivateVdsVDSCommand, log id: 3bd8510d >2013-03-11 00:40:00,302 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-86) [1c9de79] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:40:00,303 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-86) [1c9de79] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 00:40:00,304 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [1c9de79] Autorecovering 1 storage domains >2013-03-11 00:40:00,304 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [1c9de79] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 00:40:00,305 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-86) [495520dc] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:40:00,305 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-86) [495520dc] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:40 AM >2013-03-11 00:40:00,308 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-86) [495520dc] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:40 AM >2013-03-11 00:40:02,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-6) [cbde079] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1efba3 >2013-03-11 00:40:02,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-6) [cbde079] FINISH, GetHardwareInfoVDSCommand, log id: 1efba3 >2013-03-11 00:40:02,069 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-6) [1983d41f] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 00:40:02,073 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-6) [22887614] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 00:40:02,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5e7b4904 >2013-03-11 00:40:08,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 00:40:08,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:40:08,069 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 00:40:08,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 5e7b4904 >2013-03-11 00:40:08,073 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [72f420f2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:40:08,087 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [72f420f2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 56d9549 >2013-03-11 00:40:08,087 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [72f420f2] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 00:40:08,087 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [72f420f2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 56d9549 >2013-03-11 00:40:08,089 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [72f420f2] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 00:40:08,089 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [22887614] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 00:40:08,100 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-6) [35cc263c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:40:08,101 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-6) [35cc263c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5d5ecb2e >2013-03-11 00:40:08,103 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-6) [35cc263c] FINISH, SetVdsStatusVDSCommand, log id: 5d5ecb2e >2013-03-11 00:40:08,110 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-6) [35cc263c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 00:40:08,134 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-6) [cd3ff39] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:40:08,136 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-6) [6d119077] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:40:08,138 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-6) [6d119077] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 00:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [b10032] Autorecovering 1 hosts >2013-03-11 00:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [b10032] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [2922858] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [2922858] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [2922858] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 327e7993 >2013-03-11 00:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [2922858] FINISH, SetVdsStatusVDSCommand, log id: 327e7993 >2013-03-11 00:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [2922858] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1ba874c5 >2013-03-11 00:45:00,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [2922858] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7a538214 >2013-03-11 00:45:00,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [2922858] FINISH, GetHardwareInfoVDSCommand, log id: 7a538214 >2013-03-11 00:45:00,325 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [3d9522da] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:45:00,328 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [360a4c26] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:45:00,329 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [360a4c26] FINISH, ActivateVdsVDSCommand, log id: 1ba874c5 >2013-03-11 00:45:00,349 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [360a4c26] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:45:00,349 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [360a4c26] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 00:45:00,351 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [360a4c26] Autorecovering 1 storage domains >2013-03-11 00:45:00,351 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [360a4c26] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 00:45:00,351 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [5cd38640] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:45:00,351 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [5cd38640] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:45 AM >2013-03-11 00:45:00,354 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [5cd38640] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:45 AM >2013-03-11 00:45:02,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [111d3efa] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 174ded22 >2013-03-11 00:45:02,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [111d3efa] FINISH, GetHardwareInfoVDSCommand, log id: 174ded22 >2013-03-11 00:45:02,420 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-54) [1326d78f] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 00:45:02,423 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-54) [706c2367] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 00:45:02,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5082984 >2013-03-11 00:45:09,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 00:45:09,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:45:09,406 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 00:45:09,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 5082984 >2013-03-11 00:45:09,410 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [147ad294] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:45:09,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [147ad294] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 693a879c >2013-03-11 00:45:09,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [147ad294] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 00:45:09,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [147ad294] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 693a879c >2013-03-11 00:45:09,426 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [147ad294] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 00:45:09,426 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [706c2367] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 00:45:09,437 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-54) [6607786e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:45:09,438 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [6607786e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: a1d408a >2013-03-11 00:45:09,439 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [6607786e] FINISH, SetVdsStatusVDSCommand, log id: a1d408a >2013-03-11 00:45:09,446 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [6607786e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 00:45:09,470 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [230bf80e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:45:09,473 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [2a634207] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:45:09,474 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-54) [2a634207] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 00:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [12a1573c] Autorecovering 1 hosts >2013-03-11 00:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [12a1573c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [450c21d5] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [450c21d5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:50:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [450c21d5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 57fb5fd5 >2013-03-11 00:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [450c21d5] FINISH, SetVdsStatusVDSCommand, log id: 57fb5fd5 >2013-03-11 00:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-64) [450c21d5] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1379f9d3 >2013-03-11 00:50:00,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [450c21d5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 71a84372 >2013-03-11 00:50:00,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [450c21d5] FINISH, GetHardwareInfoVDSCommand, log id: 71a84372 >2013-03-11 00:50:00,786 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-64) [4bc626af] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:50:00,795 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-64) [58a87dc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:50:00,797 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-64) [58a87dc] FINISH, ActivateVdsVDSCommand, log id: 1379f9d3 >2013-03-11 00:50:00,818 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [58a87dc] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:50:00,819 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-64) [58a87dc] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 00:50:00,820 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [58a87dc] Autorecovering 1 storage domains >2013-03-11 00:50:00,820 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [58a87dc] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 00:50:00,821 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [f501cee] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:50:00,821 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [f501cee] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:50 AM >2013-03-11 00:50:00,823 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [f501cee] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:50 AM >2013-03-11 00:50:00,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-85) [308d93ab] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1be1a789 >2013-03-11 00:50:00,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-85) [308d93ab] FINISH, GetHardwareInfoVDSCommand, log id: 1be1a789 >2013-03-11 00:50:00,903 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-85) [2d71a25f] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 00:50:00,906 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-85) [a92c280] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 00:50:00,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 52add601 >2013-03-11 00:50:06,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 00:50:06,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:50:06,885 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 00:50:06,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 52add601 >2013-03-11 00:50:06,889 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [39769d6a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:50:06,902 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [39769d6a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 789502e0 >2013-03-11 00:50:06,902 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [39769d6a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 00:50:06,902 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [39769d6a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 789502e0 >2013-03-11 00:50:06,904 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [39769d6a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 00:50:06,905 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [a92c280] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 00:50:06,914 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-85) [22313e5b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:50:06,916 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-85) [22313e5b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7e4db306 >2013-03-11 00:50:06,917 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-85) [22313e5b] FINISH, SetVdsStatusVDSCommand, log id: 7e4db306 >2013-03-11 00:50:06,923 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-85) [22313e5b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 00:50:06,946 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-85) [53779890] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:50:06,949 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-85) [5f78ab61] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:50:06,950 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-85) [5f78ab61] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 00:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [2201893f] Autorecovering 1 hosts >2013-03-11 00:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [2201893f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [6c92dad2] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [6c92dad2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [6c92dad2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: c74ffa4 >2013-03-11 00:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [6c92dad2] FINISH, SetVdsStatusVDSCommand, log id: c74ffa4 >2013-03-11 00:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-46) [6c92dad2] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 404c590e >2013-03-11 00:55:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [6c92dad2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3adfc879 >2013-03-11 00:55:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [6c92dad2] FINISH, GetHardwareInfoVDSCommand, log id: 3adfc879 >2013-03-11 00:55:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-46) [6c6c945b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:55:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-46) [3d4a5c9d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:55:00,093 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-46) [3d4a5c9d] FINISH, ActivateVdsVDSCommand, log id: 404c590e >2013-03-11 00:55:00,112 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [3d4a5c9d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 00:55:00,113 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-46) [3d4a5c9d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 00:55:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [3d4a5c9d] Autorecovering 1 storage domains >2013-03-11 00:55:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [3d4a5c9d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 00:55:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [612d1d72] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:55:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [612d1d72] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:55 AM >2013-03-11 00:55:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [612d1d72] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:55 AM >2013-03-11 00:55:01,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [2ee33b29] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: ccecd96 >2013-03-11 00:55:01,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [2ee33b29] FINISH, GetHardwareInfoVDSCommand, log id: ccecd96 >2013-03-11 00:55:01,232 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-44) [7f96b084] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 00:55:01,235 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-44) [43665736] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 00:55:01,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3f28d3ca >2013-03-11 00:55:04,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 00:55:04,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 00:55:04,223 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 00:55:04,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 3f28d3ca >2013-03-11 00:55:04,227 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [2ff2befe] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 00:55:04,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [2ff2befe] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 68c0dd60 >2013-03-11 00:55:04,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [2ff2befe] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 00:55:04,241 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [2ff2befe] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 68c0dd60 >2013-03-11 00:55:04,243 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [2ff2befe] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 00:55:04,243 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [43665736] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 00:55:04,253 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-44) [1a69eba0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:55:04,255 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [1a69eba0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 712470a7 >2013-03-11 00:55:04,257 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [1a69eba0] FINISH, SetVdsStatusVDSCommand, log id: 712470a7 >2013-03-11 00:55:04,263 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-44) [1a69eba0] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 00:55:04,285 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-44) [694a533a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:55:04,287 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-44) [36b64128] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 00:55:04,289 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-44) [36b64128] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 01:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [739bd752] Autorecovering 1 hosts >2013-03-11 01:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [739bd752] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [650afed2] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [650afed2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:00:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [650afed2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 16319747 >2013-03-11 01:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [650afed2] FINISH, SetVdsStatusVDSCommand, log id: 16319747 >2013-03-11 01:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-57) [650afed2] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 13ddbfe2 >2013-03-11 01:00:00,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [650afed2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 51697508 >2013-03-11 01:00:00,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [650afed2] FINISH, GetHardwareInfoVDSCommand, log id: 51697508 >2013-03-11 01:00:00,100 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-57) [7645ed01] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:00:00,103 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-57) [1ec32b67] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:00:00,104 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-57) [1ec32b67] FINISH, ActivateVdsVDSCommand, log id: 13ddbfe2 >2013-03-11 01:00:00,123 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [1ec32b67] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:00:00,124 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-57) [1ec32b67] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 01:00:00,125 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [1ec32b67] Autorecovering 1 storage domains >2013-03-11 01:00:00,125 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [1ec32b67] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 01:00:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [51e25f53] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:00:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [51e25f53] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:00 AM >2013-03-11 01:00:00,129 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [51e25f53] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:00 AM >2013-03-11 01:00:01,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [1e5defb2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 51782e90 >2013-03-11 01:00:01,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [1e5defb2] FINISH, GetHardwareInfoVDSCommand, log id: 51782e90 >2013-03-11 01:00:01,574 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-90) [2e3ffa4] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 01:00:01,578 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-90) [4cc05780] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 01:00:01,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4f1d799d >2013-03-11 01:00:07,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 01:00:07,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:00:07,623 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 01:00:07,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 4f1d799d >2013-03-11 01:00:07,627 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [45456df7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:00:07,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [45456df7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1074d6ae >2013-03-11 01:00:07,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [45456df7] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 01:00:07,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [45456df7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1074d6ae >2013-03-11 01:00:07,643 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [45456df7] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 01:00:07,643 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [4cc05780] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 01:00:07,653 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-90) [269476dc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:00:07,654 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [269476dc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 281eb91c >2013-03-11 01:00:07,655 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [269476dc] FINISH, SetVdsStatusVDSCommand, log id: 281eb91c >2013-03-11 01:00:07,661 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [269476dc] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 01:00:07,683 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-90) [ae28fb3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:00:07,685 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-90) [22d05a6a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:00:07,687 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-90) [22d05a6a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 01:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [4e3ff719] Autorecovering 1 hosts >2013-03-11 01:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [4e3ff719] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [71253a07] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [71253a07] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [71253a07] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 57bef3d >2013-03-11 01:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [71253a07] FINISH, SetVdsStatusVDSCommand, log id: 57bef3d >2013-03-11 01:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [71253a07] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 33861f00 >2013-03-11 01:05:00,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [71253a07] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: e99b85a >2013-03-11 01:05:00,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [71253a07] FINISH, GetHardwareInfoVDSCommand, log id: e99b85a >2013-03-11 01:05:00,246 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-16) [456ffd6f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:05:00,248 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-16) [752d14c9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:05:00,249 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [752d14c9] FINISH, ActivateVdsVDSCommand, log id: 33861f00 >2013-03-11 01:05:00,268 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [752d14c9] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:05:00,269 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-16) [752d14c9] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 01:05:00,270 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [752d14c9] Autorecovering 1 storage domains >2013-03-11 01:05:00,271 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [752d14c9] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 01:05:00,271 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [1d1d7172] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:05:00,271 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [1d1d7172] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:05 AM >2013-03-11 01:05:00,274 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [1d1d7172] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:05 AM >2013-03-11 01:05:01,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [5cceb414] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 744017ed >2013-03-11 01:05:01,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [5cceb414] FINISH, GetHardwareInfoVDSCommand, log id: 744017ed >2013-03-11 01:05:01,968 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-24) [7bb0b863] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 01:05:01,971 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-24) [548737c8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 01:05:01,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 35ab8528 >2013-03-11 01:05:08,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 01:05:08,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:05:08,995 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 01:05:08,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 35ab8528 >2013-03-11 01:05:08,998 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [3364e66c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:05:09,012 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [3364e66c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2decfd53 >2013-03-11 01:05:09,012 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [3364e66c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 01:05:09,012 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [3364e66c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2decfd53 >2013-03-11 01:05:09,014 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [3364e66c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 01:05:09,015 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [548737c8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 01:05:09,024 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-24) [53f1c501] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:05:09,026 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [53f1c501] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 41c92908 >2013-03-11 01:05:09,027 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [53f1c501] FINISH, SetVdsStatusVDSCommand, log id: 41c92908 >2013-03-11 01:05:09,033 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-24) [53f1c501] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 01:05:09,055 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-24) [ad2d96c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:05:09,058 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-24) [3268391f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:05:09,059 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-24) [3268391f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 01:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [2c221c0b] Autorecovering 1 hosts >2013-03-11 01:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [2c221c0b] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [43f9a002] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [43f9a002] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [43f9a002] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 699c7900 >2013-03-11 01:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [43f9a002] FINISH, SetVdsStatusVDSCommand, log id: 699c7900 >2013-03-11 01:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [43f9a002] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 555c88c >2013-03-11 01:10:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [43f9a002] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 598651ac >2013-03-11 01:10:00,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [43f9a002] FINISH, GetHardwareInfoVDSCommand, log id: 598651ac >2013-03-11 01:10:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [232cf4e4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:10:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [434ad49c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:10:00,205 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [434ad49c] FINISH, ActivateVdsVDSCommand, log id: 555c88c >2013-03-11 01:10:00,224 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [434ad49c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:10:00,225 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [434ad49c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 01:10:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [434ad49c] Autorecovering 1 storage domains >2013-03-11 01:10:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [434ad49c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 01:10:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [585ec7ac] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:10:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [585ec7ac] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:10 AM >2013-03-11 01:10:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [585ec7ac] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:10 AM >2013-03-11 01:10:00,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [7b1a4385] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 278f50fe >2013-03-11 01:10:00,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [7b1a4385] FINISH, GetHardwareInfoVDSCommand, log id: 278f50fe >2013-03-11 01:10:00,338 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-48) [3f21a6f0] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 01:10:00,341 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-48) [76880968] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 01:10:00,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 111a76f8 >2013-03-11 01:10:08,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 01:10:08,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:10:08,350 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 01:10:08,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 111a76f8 >2013-03-11 01:10:08,353 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [67fa269e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:10:08,367 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [67fa269e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 37eb33b6 >2013-03-11 01:10:08,367 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [67fa269e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 01:10:08,367 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [67fa269e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 37eb33b6 >2013-03-11 01:10:08,369 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [67fa269e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 01:10:08,370 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [76880968] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 01:10:08,379 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-48) [32b43562] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:10:08,380 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [32b43562] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 666625e7 >2013-03-11 01:10:08,382 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [32b43562] FINISH, SetVdsStatusVDSCommand, log id: 666625e7 >2013-03-11 01:10:08,387 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-48) [32b43562] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 01:10:08,427 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-48) [36a8eb73] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:10:08,430 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-48) [347a0f47] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:10:08,431 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-48) [347a0f47] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 01:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [e4e30f] Autorecovering 1 hosts >2013-03-11 01:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [e4e30f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-80) [60c2f728] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-80) [60c2f728] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [60c2f728] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3dd0220 >2013-03-11 01:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [60c2f728] FINISH, SetVdsStatusVDSCommand, log id: 3dd0220 >2013-03-11 01:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-80) [60c2f728] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 236c579d >2013-03-11 01:15:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [60c2f728] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 66851443 >2013-03-11 01:15:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [60c2f728] FINISH, GetHardwareInfoVDSCommand, log id: 66851443 >2013-03-11 01:15:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-80) [637e4608] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:15:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-80) [6ff66e78] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:15:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-80) [6ff66e78] FINISH, ActivateVdsVDSCommand, log id: 236c579d >2013-03-11 01:15:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-80) [6ff66e78] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:15:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-80) [6ff66e78] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 01:15:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [6ff66e78] Autorecovering 1 storage domains >2013-03-11 01:15:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [6ff66e78] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 01:15:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-80) [7cc06508] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:15:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-80) [7cc06508] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:15 AM >2013-03-11 01:15:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-80) [7cc06508] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:15 AM >2013-03-11 01:15:02,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [675dc8bd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 47cd1328 >2013-03-11 01:15:02,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [675dc8bd] FINISH, GetHardwareInfoVDSCommand, log id: 47cd1328 >2013-03-11 01:15:02,717 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-66) [7a308395] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 01:15:02,720 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-66) [43c01b60] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 01:15:02,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6bf76154 >2013-03-11 01:15:08,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 01:15:08,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:15:08,702 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 01:15:08,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 6bf76154 >2013-03-11 01:15:08,705 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [35a0471d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:15:08,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [35a0471d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2ddb2864 >2013-03-11 01:15:08,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [35a0471d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 01:15:08,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [35a0471d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2ddb2864 >2013-03-11 01:15:08,721 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [35a0471d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 01:15:08,722 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [43c01b60] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 01:15:08,731 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-66) [69d3ebac] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:15:08,733 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [69d3ebac] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 75f34a1a >2013-03-11 01:15:08,734 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [69d3ebac] FINISH, SetVdsStatusVDSCommand, log id: 75f34a1a >2013-03-11 01:15:08,739 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-66) [69d3ebac] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 01:15:08,762 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-66) [689c8dbd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:15:08,764 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-66) [54a19484] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:15:08,765 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-66) [54a19484] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 01:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [4a1e14e4] Autorecovering 1 hosts >2013-03-11 01:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [4a1e14e4] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:20:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [dcb4d45] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:20:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [dcb4d45] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:20:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [dcb4d45] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1e17c059 >2013-03-11 01:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [dcb4d45] FINISH, SetVdsStatusVDSCommand, log id: 1e17c059 >2013-03-11 01:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-18) [dcb4d45] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 551f2566 >2013-03-11 01:20:00,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [dcb4d45] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2b3f2e49 >2013-03-11 01:20:00,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [dcb4d45] FINISH, GetHardwareInfoVDSCommand, log id: 2b3f2e49 >2013-03-11 01:20:00,308 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [5b57ced5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:20:00,310 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [42bd886a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:20:00,312 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-18) [42bd886a] FINISH, ActivateVdsVDSCommand, log id: 551f2566 >2013-03-11 01:20:00,333 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [42bd886a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:20:00,334 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [42bd886a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 01:20:00,335 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [42bd886a] Autorecovering 1 storage domains >2013-03-11 01:20:00,335 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [42bd886a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 01:20:00,336 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [41e903d0] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:20:00,336 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [41e903d0] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:20 AM >2013-03-11 01:20:00,339 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [41e903d0] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:20 AM >2013-03-11 01:20:02,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [50577c89] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2b9ca381 >2013-03-11 01:20:03,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [50577c89] FINISH, GetHardwareInfoVDSCommand, log id: 2b9ca381 >2013-03-11 01:20:03,053 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-31) [274c9f24] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 01:20:03,056 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-31) [1fd1daba] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 01:20:03,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 55bd0c5 >2013-03-11 01:20:09,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 01:20:09,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:20:09,029 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 01:20:09,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 55bd0c5 >2013-03-11 01:20:09,033 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [59bcfa2c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:20:09,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [59bcfa2c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 10aff785 >2013-03-11 01:20:09,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [59bcfa2c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 01:20:09,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [59bcfa2c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 10aff785 >2013-03-11 01:20:09,048 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [59bcfa2c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 01:20:09,049 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [1fd1daba] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 01:20:09,058 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-31) [7c531579] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:20:09,059 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [7c531579] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 43c39319 >2013-03-11 01:20:09,060 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [7c531579] FINISH, SetVdsStatusVDSCommand, log id: 43c39319 >2013-03-11 01:20:09,066 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-31) [7c531579] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 01:20:09,089 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-31) [619dbca7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:20:09,091 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-31) [70b744d8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:20:09,093 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-31) [70b744d8] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 01:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [30e5b0bc] Autorecovering 1 hosts >2013-03-11 01:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [30e5b0bc] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-4) [51c3d9a2] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-4) [51c3d9a2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:25:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [51c3d9a2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 445f6c27 >2013-03-11 01:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [51c3d9a2] FINISH, SetVdsStatusVDSCommand, log id: 445f6c27 >2013-03-11 01:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-4) [51c3d9a2] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6b25f0c7 >2013-03-11 01:25:00,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [51c3d9a2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1be48cb0 >2013-03-11 01:25:00,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [51c3d9a2] FINISH, GetHardwareInfoVDSCommand, log id: 1be48cb0 >2013-03-11 01:25:00,448 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-4) [436885dd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:25:00,450 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-4) [536d63df] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:25:00,452 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-4) [536d63df] FINISH, ActivateVdsVDSCommand, log id: 6b25f0c7 >2013-03-11 01:25:00,471 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-4) [536d63df] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:25:00,472 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-4) [536d63df] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 01:25:00,473 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [536d63df] Autorecovering 1 storage domains >2013-03-11 01:25:00,473 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [536d63df] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 01:25:00,474 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-4) [c7e06fa] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:25:00,474 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-4) [c7e06fa] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:25 AM >2013-03-11 01:25:00,477 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-4) [c7e06fa] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:25 AM >2013-03-11 01:25:00,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [b5dcd2e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 25100130 >2013-03-11 01:25:00,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [b5dcd2e] FINISH, GetHardwareInfoVDSCommand, log id: 25100130 >2013-03-11 01:25:00,552 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-83) [7803cc2c] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 01:25:00,555 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-83) [58bd253e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 01:25:00,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1a68d82d >2013-03-11 01:25:06,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 01:25:06,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:25:06,547 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 01:25:06,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 1a68d82d >2013-03-11 01:25:06,551 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [463a6809] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:25:06,565 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [463a6809] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 30f1ff22 >2013-03-11 01:25:06,565 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [463a6809] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 01:25:06,565 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [463a6809] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 30f1ff22 >2013-03-11 01:25:06,570 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [58bd253e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 01:25:06,579 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-83) [2949f19b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:25:06,581 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [2949f19b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: cde1a07 >2013-03-11 01:25:06,582 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [2949f19b] FINISH, SetVdsStatusVDSCommand, log id: cde1a07 >2013-03-11 01:25:06,588 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-83) [2949f19b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 01:25:06,610 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-83) [20f83795] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:25:06,613 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-83) [b903631] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:25:06,614 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-83) [b903631] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 01:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [353b2ed0] Autorecovering 1 hosts >2013-03-11 01:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [353b2ed0] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [5917325d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [5917325d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [5917325d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6df44b58 >2013-03-11 01:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [5917325d] FINISH, SetVdsStatusVDSCommand, log id: 6df44b58 >2013-03-11 01:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-89) [5917325d] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 12a6f595 >2013-03-11 01:30:00,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [5917325d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 72da5b4 >2013-03-11 01:30:00,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [5917325d] FINISH, GetHardwareInfoVDSCommand, log id: 72da5b4 >2013-03-11 01:30:00,275 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-89) [7934711e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:30:00,278 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-89) [111296dd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:30:00,279 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-89) [111296dd] FINISH, ActivateVdsVDSCommand, log id: 12a6f595 >2013-03-11 01:30:00,298 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [111296dd] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:30:00,299 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-89) [111296dd] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 01:30:00,300 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [111296dd] Autorecovering 1 storage domains >2013-03-11 01:30:00,300 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [111296dd] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 01:30:00,301 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [3fb0fe59] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:30:00,301 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [3fb0fe59] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:30 AM >2013-03-11 01:30:00,304 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [3fb0fe59] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:30 AM >2013-03-11 01:30:00,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [6ff7b60c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 69d95520 >2013-03-11 01:30:00,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [6ff7b60c] FINISH, GetHardwareInfoVDSCommand, log id: 69d95520 >2013-03-11 01:30:00,897 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-79) [22fb26c7] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 01:30:00,900 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-79) [45be86f8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 01:30:00,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 37341c80 >2013-03-11 01:30:08,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 01:30:08,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:30:08,926 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 01:30:08,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 37341c80 >2013-03-11 01:30:08,930 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [71da5327] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:30:08,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [71da5327] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 45a9a258 >2013-03-11 01:30:08,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [71da5327] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 01:30:08,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [71da5327] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 45a9a258 >2013-03-11 01:30:08,945 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [71da5327] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 01:30:08,946 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [45be86f8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 01:30:08,955 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-79) [714b6925] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:30:08,957 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [714b6925] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6f3fdfba >2013-03-11 01:30:08,958 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [714b6925] FINISH, SetVdsStatusVDSCommand, log id: 6f3fdfba >2013-03-11 01:30:08,964 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-79) [714b6925] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 01:30:08,986 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-79) [17cf7fcb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:30:08,989 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-79) [322cd7cc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:30:08,990 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-79) [322cd7cc] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 01:30:33,864 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-99) [6eef8afb] Start refreshing all users data >2013-03-11 01:30:35,041 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-55) [1c38d809] Attempting to update VMs/Templates Ovf. >2013-03-11 01:30:35,042 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-55) [1c38d809] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 01:30:35,043 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-55) [1c38d809] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 01:30:35,043 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-55) [1c38d809] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 01:30:35,043 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-55) [1c38d809] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 01:30:35,043 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-55) [1c38d809] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 01:30:35,044 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-55) [1c38d809] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 01:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [57b553a4] Autorecovering 1 hosts >2013-03-11 01:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [57b553a4] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [6cb1ad45] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [6cb1ad45] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [6cb1ad45] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 42147767 >2013-03-11 01:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [6cb1ad45] FINISH, SetVdsStatusVDSCommand, log id: 42147767 >2013-03-11 01:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [6cb1ad45] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6ecfe661 >2013-03-11 01:35:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [6cb1ad45] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 11d2055a >2013-03-11 01:35:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [6cb1ad45] FINISH, GetHardwareInfoVDSCommand, log id: 11d2055a >2013-03-11 01:35:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-63) [11f0f4ee] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:35:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-63) [3489f3de] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:35:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [3489f3de] FINISH, ActivateVdsVDSCommand, log id: 6ecfe661 >2013-03-11 01:35:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [3489f3de] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:35:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-63) [3489f3de] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 01:35:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [3489f3de] Autorecovering 1 storage domains >2013-03-11 01:35:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [3489f3de] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 01:35:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [7e19e9ff] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:35:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [7e19e9ff] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:35 AM >2013-03-11 01:35:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [7e19e9ff] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:35 AM >2013-03-11 01:35:00,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [41e903d0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3d3cdd30 >2013-03-11 01:35:00,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [41e903d0] FINISH, GetHardwareInfoVDSCommand, log id: 3d3cdd30 >2013-03-11 01:35:00,298 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-18) [1d0ea97e] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 01:35:00,301 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-18) [41674ba0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 01:35:00,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: b4a6c9d >2013-03-11 01:35:07,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 01:35:07,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:35:07,629 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 01:35:07,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: b4a6c9d >2013-03-11 01:35:07,633 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [3295aafa] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:35:07,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [3295aafa] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 93c3b68 >2013-03-11 01:35:07,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [3295aafa] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 01:35:07,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [3295aafa] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 93c3b68 >2013-03-11 01:35:07,648 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [3295aafa] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 01:35:07,649 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [41674ba0] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 01:35:07,658 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-18) [4fa42764] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:35:07,659 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [4fa42764] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 233a2f1d >2013-03-11 01:35:07,661 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [4fa42764] FINISH, SetVdsStatusVDSCommand, log id: 233a2f1d >2013-03-11 01:35:07,666 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [4fa42764] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 01:35:07,698 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [7977fffb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:35:07,700 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [659d7a3f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:35:07,701 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) [659d7a3f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 01:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [60227064] Autorecovering 1 hosts >2013-03-11 01:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [60227064] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [36440aeb] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [36440aeb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [36440aeb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5c6b89b1 >2013-03-11 01:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [36440aeb] FINISH, SetVdsStatusVDSCommand, log id: 5c6b89b1 >2013-03-11 01:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [36440aeb] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 319ad4e >2013-03-11 01:40:00,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [36440aeb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 735ebb25 >2013-03-11 01:40:00,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [36440aeb] FINISH, GetHardwareInfoVDSCommand, log id: 735ebb25 >2013-03-11 01:40:00,208 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-69) [82462d1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:40:00,210 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-69) [6bff687a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:40:00,212 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [6bff687a] FINISH, ActivateVdsVDSCommand, log id: 319ad4e >2013-03-11 01:40:00,231 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [6bff687a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:40:00,232 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-69) [6bff687a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 01:40:00,233 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [6bff687a] Autorecovering 1 storage domains >2013-03-11 01:40:00,233 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [6bff687a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 01:40:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [35c5f85c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:40:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [35c5f85c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:40 AM >2013-03-11 01:40:00,237 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [35c5f85c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:40 AM >2013-03-11 01:40:01,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [585ec7ac] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 220eb12 >2013-03-11 01:40:01,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [585ec7ac] FINISH, GetHardwareInfoVDSCommand, log id: 220eb12 >2013-03-11 01:40:01,978 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-17) [76899f37] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 01:40:01,982 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-17) [3a6cfc36] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 01:40:01,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3b4aa6b1 >2013-03-11 01:40:08,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 01:40:08,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:40:08,971 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 01:40:08,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 3b4aa6b1 >2013-03-11 01:40:08,974 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [975b7ac] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:40:08,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [975b7ac] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1cf565f8 >2013-03-11 01:40:08,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [975b7ac] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 01:40:08,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [975b7ac] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1cf565f8 >2013-03-11 01:40:08,990 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [975b7ac] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 01:40:08,991 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [3a6cfc36] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 01:40:09,000 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-17) [5fe670ae] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:40:09,002 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [5fe670ae] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4b2fdf7d >2013-03-11 01:40:09,003 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [5fe670ae] FINISH, SetVdsStatusVDSCommand, log id: 4b2fdf7d >2013-03-11 01:40:09,009 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [5fe670ae] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 01:40:09,030 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [1396249a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:40:09,033 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [58333aca] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:40:09,034 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-17) [58333aca] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 01:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [6aa6891a] Autorecovering 1 hosts >2013-03-11 01:45:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [6aa6891a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [1d350984] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [1d350984] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [1d350984] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6db89109 >2013-03-11 01:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [1d350984] FINISH, SetVdsStatusVDSCommand, log id: 6db89109 >2013-03-11 01:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-77) [1d350984] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4eacad64 >2013-03-11 01:45:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [1d350984] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: f3f60f1 >2013-03-11 01:45:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [1d350984] FINISH, GetHardwareInfoVDSCommand, log id: f3f60f1 >2013-03-11 01:45:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-77) [7da10104] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:45:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-77) [d9ed4c1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:45:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-77) [d9ed4c1] FINISH, ActivateVdsVDSCommand, log id: 4eacad64 >2013-03-11 01:45:00,222 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [d9ed4c1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:45:00,227 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-77) [d9ed4c1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 01:45:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [d9ed4c1] Autorecovering 1 storage domains >2013-03-11 01:45:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [d9ed4c1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 01:45:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [337c7da4] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:45:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [337c7da4] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:45 AM >2013-03-11 01:45:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [337c7da4] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:45 AM >2013-03-11 01:45:00,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [3fb0fe59] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 28d997dc >2013-03-11 01:45:00,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [3fb0fe59] FINISH, GetHardwareInfoVDSCommand, log id: 28d997dc >2013-03-11 01:45:00,314 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-89) [4a233890] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 01:45:00,317 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-89) [1e94d16d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 01:45:00,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2d2ce04d >2013-03-11 01:45:05,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 01:45:05,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:45:05,299 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 01:45:05,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 2d2ce04d >2013-03-11 01:45:05,303 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [7710c9fa] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:45:05,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [7710c9fa] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 66c1515d >2013-03-11 01:45:05,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [7710c9fa] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 01:45:05,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [7710c9fa] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 66c1515d >2013-03-11 01:45:05,318 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [7710c9fa] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 01:45:05,319 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [1e94d16d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 01:45:05,328 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-89) [3cff7a5e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:45:05,329 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [3cff7a5e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5f1a6d9b >2013-03-11 01:45:05,331 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [3cff7a5e] FINISH, SetVdsStatusVDSCommand, log id: 5f1a6d9b >2013-03-11 01:45:05,336 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-89) [3cff7a5e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 01:45:05,359 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-89) [37da29c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:45:05,362 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-89) [28cf91a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:45:05,363 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-89) [28cf91a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 01:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [3d67cecb] Autorecovering 1 hosts >2013-03-11 01:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [3d67cecb] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-74) [121abfe1] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-74) [121abfe1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [121abfe1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7950ad62 >2013-03-11 01:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [121abfe1] FINISH, SetVdsStatusVDSCommand, log id: 7950ad62 >2013-03-11 01:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-74) [121abfe1] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 385cde89 >2013-03-11 01:50:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [121abfe1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 396924a4 >2013-03-11 01:50:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [121abfe1] FINISH, GetHardwareInfoVDSCommand, log id: 396924a4 >2013-03-11 01:50:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-74) [50ac7114] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:50:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-74) [5fde42c0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:50:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-74) [5fde42c0] FINISH, ActivateVdsVDSCommand, log id: 385cde89 >2013-03-11 01:50:00,219 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-74) [5fde42c0] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:50:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-74) [5fde42c0] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 01:50:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [5fde42c0] Autorecovering 1 storage domains >2013-03-11 01:50:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [5fde42c0] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 01:50:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-74) [124dcd67] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:50:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-74) [124dcd67] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:50 AM >2013-03-11 01:50:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-74) [124dcd67] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:50 AM >2013-03-11 01:50:02,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [1d64616f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4d079318 >2013-03-11 01:50:02,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [1d64616f] FINISH, GetHardwareInfoVDSCommand, log id: 4d079318 >2013-03-11 01:50:02,650 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-25) [6bc9a1b6] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 01:50:02,653 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-25) [1fb358b1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 01:50:02,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5a4a94f6 >2013-03-11 01:50:08,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 01:50:08,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:50:08,662 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 01:50:08,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 5a4a94f6 >2013-03-11 01:50:08,667 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [76d71c27] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:50:08,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [76d71c27] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 242c4bf5 >2013-03-11 01:50:08,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [76d71c27] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 01:50:08,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [76d71c27] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 242c4bf5 >2013-03-11 01:50:08,682 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [76d71c27] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 01:50:08,683 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [1fb358b1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 01:50:08,692 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-25) [11f4e246] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:50:08,693 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [11f4e246] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: a14de62 >2013-03-11 01:50:08,694 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [11f4e246] FINISH, SetVdsStatusVDSCommand, log id: a14de62 >2013-03-11 01:50:08,700 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-25) [11f4e246] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 01:50:08,723 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-25) [3062a2c1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:50:08,726 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-25) [d80ce96] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:50:08,727 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-25) [d80ce96] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 01:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) [3e9fbe] Autorecovering 1 hosts >2013-03-11 01:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) [3e9fbe] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-30) [45802f4f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-30) [45802f4f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-30) [45802f4f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7953f76f >2013-03-11 01:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-30) [45802f4f] FINISH, SetVdsStatusVDSCommand, log id: 7953f76f >2013-03-11 01:55:00,010 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-30) [45802f4f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2199f12d >2013-03-11 01:55:00,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-30) [45802f4f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 997273d >2013-03-11 01:55:00,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-30) [45802f4f] FINISH, GetHardwareInfoVDSCommand, log id: 997273d >2013-03-11 01:55:00,194 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-30) [624c194b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:55:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-30) [32aa8e5e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:55:00,198 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-30) [32aa8e5e] FINISH, ActivateVdsVDSCommand, log id: 2199f12d >2013-03-11 01:55:00,218 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-30) [32aa8e5e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 01:55:00,219 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-30) [32aa8e5e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 01:55:00,220 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) [32aa8e5e] Autorecovering 1 storage domains >2013-03-11 01:55:00,220 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) [32aa8e5e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 01:55:00,221 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-30) [7675f75] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:55:00,221 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-30) [7675f75] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:55 AM >2013-03-11 01:55:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-30) [7675f75] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:55 AM >2013-03-11 01:55:02,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [c7e06fa] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 209952a5 >2013-03-11 01:55:02,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [c7e06fa] FINISH, GetHardwareInfoVDSCommand, log id: 209952a5 >2013-03-11 01:55:03,006 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-4) [22030a54] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 01:55:03,012 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-4) [415d0ea1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 01:55:03,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 106d2791 >2013-03-11 01:55:07,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 01:55:07,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 01:55:07,990 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 01:55:07,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 106d2791 >2013-03-11 01:55:07,994 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [677c4494] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 01:55:08,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [677c4494] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c343917 >2013-03-11 01:55:08,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [677c4494] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 01:55:08,008 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [677c4494] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c343917 >2013-03-11 01:55:08,012 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [677c4494] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 01:55:08,013 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [415d0ea1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 01:55:08,023 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-4) [561b5414] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:55:08,024 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [561b5414] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 724fd10f >2013-03-11 01:55:08,026 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [561b5414] FINISH, SetVdsStatusVDSCommand, log id: 724fd10f >2013-03-11 01:55:08,032 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-4) [561b5414] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 01:55:08,054 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-4) [11f9c372] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:55:08,056 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-4) [71240b2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 01:55:08,058 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-4) [71240b2] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 02:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [37e37665] Autorecovering 1 hosts >2013-03-11 02:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [37e37665] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-98) [5c28f81b] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-98) [5c28f81b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-98) [5c28f81b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: b52fffc >2013-03-11 02:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-98) [5c28f81b] FINISH, SetVdsStatusVDSCommand, log id: b52fffc >2013-03-11 02:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-98) [5c28f81b] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6fcfbcb3 >2013-03-11 02:00:00,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-98) [5c28f81b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 757f9949 >2013-03-11 02:00:00,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-98) [5c28f81b] FINISH, GetHardwareInfoVDSCommand, log id: 757f9949 >2013-03-11 02:00:00,210 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-98) [6197c3b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:00:00,212 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-98) [1a3c3fd7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:00:00,213 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-98) [1a3c3fd7] FINISH, ActivateVdsVDSCommand, log id: 6fcfbcb3 >2013-03-11 02:00:00,233 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-98) [1a3c3fd7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:00:00,234 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-98) [1a3c3fd7] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 02:00:00,235 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [1a3c3fd7] Autorecovering 1 storage domains >2013-03-11 02:00:00,235 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [1a3c3fd7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 02:00:00,236 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-98) [260a8c0d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:00:00,236 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-98) [260a8c0d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:00 AM >2013-03-11 02:00:00,238 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-98) [260a8c0d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:00 AM >2013-03-11 02:00:02,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [2eccb85f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3afee397 >2013-03-11 02:00:02,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [2eccb85f] FINISH, GetHardwareInfoVDSCommand, log id: 3afee397 >2013-03-11 02:00:02,336 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-9) [71d6aad6] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 02:00:02,340 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-9) [344278ef] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 02:00:02,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 64bf2673 >2013-03-11 02:00:08,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 02:00:08,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:00:08,360 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 02:00:08,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 64bf2673 >2013-03-11 02:00:08,364 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [630d6cf6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:00:08,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [630d6cf6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 18583bf7 >2013-03-11 02:00:08,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [630d6cf6] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 02:00:08,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [630d6cf6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 18583bf7 >2013-03-11 02:00:08,379 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [630d6cf6] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 02:00:08,380 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [344278ef] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 02:00:08,389 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-9) [6c0ae9b7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:00:08,391 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [6c0ae9b7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 741df34d >2013-03-11 02:00:08,392 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [6c0ae9b7] FINISH, SetVdsStatusVDSCommand, log id: 741df34d >2013-03-11 02:00:08,398 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-9) [6c0ae9b7] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 02:00:08,420 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-9) [e3543c8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:00:08,423 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-9) [9046d64] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:00:08,424 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-9) [9046d64] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 02:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [38e98821] Autorecovering 1 hosts >2013-03-11 02:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [38e98821] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [2b1f1049] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [2b1f1049] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [2b1f1049] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 177278bd >2013-03-11 02:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [2b1f1049] FINISH, SetVdsStatusVDSCommand, log id: 177278bd >2013-03-11 02:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [2b1f1049] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4b749a13 >2013-03-11 02:05:00,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [2b1f1049] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 322019c5 >2013-03-11 02:05:00,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [2b1f1049] FINISH, GetHardwareInfoVDSCommand, log id: 322019c5 >2013-03-11 02:05:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [17fff9d4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:05:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [3e254ce4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:05:00,205 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [3e254ce4] FINISH, ActivateVdsVDSCommand, log id: 4b749a13 >2013-03-11 02:05:00,224 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [3e254ce4] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:05:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [3e254ce4] Autorecovering 1 storage domains >2013-03-11 02:05:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [3e254ce4] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 02:05:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [491d7e41] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:05:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [491d7e41] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:05 AM >2013-03-11 02:05:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [491d7e41] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:05 AM >2013-03-11 02:05:02,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [2bf4b7c5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6f90c1ef >2013-03-11 02:05:02,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [2bf4b7c5] FINISH, GetHardwareInfoVDSCommand, log id: 6f90c1ef >2013-03-11 02:05:02,702 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-20) [1a7ef916] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 02:05:02,705 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-20) [40c17ad6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 02:05:02,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6de7342d >2013-03-11 02:05:09,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 02:05:09,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:05:09,730 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 02:05:09,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 6de7342d >2013-03-11 02:05:09,733 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [75d2fe89] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:05:09,747 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [75d2fe89] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 561138ab >2013-03-11 02:05:09,747 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [75d2fe89] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 02:05:09,747 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [75d2fe89] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 561138ab >2013-03-11 02:05:09,749 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [75d2fe89] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 02:05:09,749 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [40c17ad6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 02:05:09,759 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-20) [617974d6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:05:09,761 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [617974d6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6c94b550 >2013-03-11 02:05:09,762 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [617974d6] FINISH, SetVdsStatusVDSCommand, log id: 6c94b550 >2013-03-11 02:05:09,768 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-20) [617974d6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 02:05:09,791 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-20) [66700d1c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:05:09,794 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-20) [5938953] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:05:09,795 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-20) [5938953] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 02:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [571972d3] Autorecovering 1 hosts >2013-03-11 02:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [571972d3] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-5) [64e32370] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-5) [64e32370] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [64e32370] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: a9ee95b >2013-03-11 02:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [64e32370] FINISH, SetVdsStatusVDSCommand, log id: a9ee95b >2013-03-11 02:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-5) [64e32370] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4ab47a7a >2013-03-11 02:10:00,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [64e32370] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 35f1a003 >2013-03-11 02:10:00,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [64e32370] FINISH, GetHardwareInfoVDSCommand, log id: 35f1a003 >2013-03-11 02:10:00,407 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-5) [227b2212] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:10:00,410 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-5) [15260cec] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:10:00,411 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-5) [15260cec] FINISH, ActivateVdsVDSCommand, log id: 4ab47a7a >2013-03-11 02:10:00,431 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-5) [15260cec] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:10:00,432 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-5) [15260cec] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 02:10:00,433 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [15260cec] Autorecovering 1 storage domains >2013-03-11 02:10:00,433 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [15260cec] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 02:10:00,434 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-5) [6dc1a565] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:10:00,434 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-5) [6dc1a565] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:10 AM >2013-03-11 02:10:00,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-5) [6dc1a565] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:10 AM >2013-03-11 02:10:01,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [6228d5ba] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 628ed90 >2013-03-11 02:10:01,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [6228d5ba] FINISH, GetHardwareInfoVDSCommand, log id: 628ed90 >2013-03-11 02:10:01,072 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-49) [19164c41] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 02:10:01,076 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-49) [78b4e611] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 02:10:01,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5ec1f9da >2013-03-11 02:10:10,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 02:10:10,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:10:10,077 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 02:10:10,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 5ec1f9da >2013-03-11 02:10:10,080 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [547d5289] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:10:10,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [547d5289] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5c60a110 >2013-03-11 02:10:10,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [547d5289] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 02:10:10,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [547d5289] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5c60a110 >2013-03-11 02:10:10,096 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [547d5289] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 02:10:10,097 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [78b4e611] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 02:10:10,106 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-49) [3c25ca65] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:10:10,107 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [3c25ca65] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 44c2e3f7 >2013-03-11 02:10:10,108 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [3c25ca65] FINISH, SetVdsStatusVDSCommand, log id: 44c2e3f7 >2013-03-11 02:10:10,113 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-49) [3c25ca65] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 02:10:10,135 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-49) [1b4209e3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:10:10,137 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-49) [21a7c7ab] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:10:10,139 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) [21a7c7ab] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 02:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [53f93e14] Autorecovering 1 hosts >2013-03-11 02:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [53f93e14] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [20312527] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [20312527] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [20312527] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 352a976f >2013-03-11 02:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [20312527] FINISH, SetVdsStatusVDSCommand, log id: 352a976f >2013-03-11 02:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [20312527] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 52656139 >2013-03-11 02:15:00,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [20312527] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 370b4638 >2013-03-11 02:15:00,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [20312527] FINISH, GetHardwareInfoVDSCommand, log id: 370b4638 >2013-03-11 02:15:00,243 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-73) [2cef85d6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:15:00,245 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-73) [325a836c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:15:00,247 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [325a836c] FINISH, ActivateVdsVDSCommand, log id: 52656139 >2013-03-11 02:15:00,266 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [325a836c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:15:00,267 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-73) [325a836c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 02:15:00,268 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [325a836c] Autorecovering 1 storage domains >2013-03-11 02:15:00,268 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [325a836c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 02:15:00,269 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [31f5325a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:15:00,269 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [31f5325a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:15 AM >2013-03-11 02:15:00,271 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [31f5325a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:15 AM >2013-03-11 02:15:01,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [f9076ac] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6ef0744d >2013-03-11 02:15:01,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [f9076ac] FINISH, GetHardwareInfoVDSCommand, log id: 6ef0744d >2013-03-11 02:15:01,409 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-32) [76025faf] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 02:15:01,413 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-32) [60756b67] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 02:15:01,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 27c60ade >2013-03-11 02:15:08,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 02:15:08,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:15:08,412 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 02:15:08,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 27c60ade >2013-03-11 02:15:08,416 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [4362edce] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:15:08,429 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [4362edce] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 50c5d3fd >2013-03-11 02:15:08,429 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [4362edce] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 02:15:08,429 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [4362edce] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 50c5d3fd >2013-03-11 02:15:08,431 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [4362edce] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 02:15:08,432 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [60756b67] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 02:15:08,441 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-32) [acb32d9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:15:08,443 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [acb32d9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2aaccc82 >2013-03-11 02:15:08,444 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [acb32d9] FINISH, SetVdsStatusVDSCommand, log id: 2aaccc82 >2013-03-11 02:15:08,450 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-32) [acb32d9] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 02:15:08,472 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-32) [583b15f3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:15:08,474 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-32) [56dad9e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:15:08,475 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-32) [56dad9e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 02:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [1faccc09] Autorecovering 1 hosts >2013-03-11 02:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [1faccc09] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [fd14e9a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [fd14e9a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:20:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [fd14e9a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 63344d2f >2013-03-11 02:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [fd14e9a] FINISH, SetVdsStatusVDSCommand, log id: 63344d2f >2013-03-11 02:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [fd14e9a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1045ee0b >2013-03-11 02:20:00,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [fd14e9a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 70ea1a33 >2013-03-11 02:20:00,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [fd14e9a] FINISH, GetHardwareInfoVDSCommand, log id: 70ea1a33 >2013-03-11 02:20:00,633 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [6b59b5bc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:20:00,636 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [55dc4e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:20:00,637 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [55dc4e] FINISH, ActivateVdsVDSCommand, log id: 1045ee0b >2013-03-11 02:20:00,657 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [55dc4e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:20:00,657 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [55dc4e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 02:20:00,659 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [55dc4e] Autorecovering 1 storage domains >2013-03-11 02:20:00,659 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [55dc4e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 02:20:00,659 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [11c2ff1d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:20:00,660 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [11c2ff1d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:20 AM >2013-03-11 02:20:00,662 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [11c2ff1d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:20 AM >2013-03-11 02:20:02,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [27c0d33b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1512395d >2013-03-11 02:20:02,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [27c0d33b] FINISH, GetHardwareInfoVDSCommand, log id: 1512395d >2013-03-11 02:20:02,758 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-34) [a56cc0a] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 02:20:02,761 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-34) [594a41ff] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 02:20:02,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 534e0a8f >2013-03-11 02:20:08,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 02:20:08,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:20:08,762 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 02:20:08,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 534e0a8f >2013-03-11 02:20:08,766 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [522e2314] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:20:08,779 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [522e2314] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5c8dc78b >2013-03-11 02:20:08,779 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [522e2314] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 02:20:08,779 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [522e2314] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5c8dc78b >2013-03-11 02:20:08,781 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [522e2314] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 02:20:08,782 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [594a41ff] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 02:20:08,791 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-34) [7d86ce09] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:20:08,793 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [7d86ce09] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 353b88fc >2013-03-11 02:20:08,794 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [7d86ce09] FINISH, SetVdsStatusVDSCommand, log id: 353b88fc >2013-03-11 02:20:08,800 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-34) [7d86ce09] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 02:20:08,823 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-34) [1a6c1c8c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:20:08,826 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-34) [1c38f254] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:20:08,827 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) [1c38f254] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 02:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [5a57de2b] Autorecovering 1 hosts >2013-03-11 02:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [5a57de2b] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-12) [68097071] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-12) [68097071] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [68097071] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2ff98f17 >2013-03-11 02:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [68097071] FINISH, SetVdsStatusVDSCommand, log id: 2ff98f17 >2013-03-11 02:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-12) [68097071] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1fc92bae >2013-03-11 02:25:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [616a5905] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7fdd63eb >2013-03-11 02:25:00,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [616a5905] FINISH, GetHardwareInfoVDSCommand, log id: 7fdd63eb >2013-03-11 02:25:00,111 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-60) [7979a56b] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 02:25:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-60) [e2f4877] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 02:25:00,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 53f31b87 >2013-03-11 02:25:06,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 02:25:06,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:25:06,108 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 02:25:06,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 53f31b87 >2013-03-11 02:25:06,112 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [40d3dbfc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:25:06,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [40d3dbfc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 45f6e1e >2013-03-11 02:25:06,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [40d3dbfc] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 02:25:06,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [40d3dbfc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 45f6e1e >2013-03-11 02:25:06,134 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [40d3dbfc] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 02:25:06,135 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [e2f4877] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 02:25:06,144 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-60) [4176f2af] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:25:06,145 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [4176f2af] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 48252e5e >2013-03-11 02:25:06,146 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [4176f2af] FINISH, SetVdsStatusVDSCommand, log id: 48252e5e >2013-03-11 02:25:06,152 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-60) [4176f2af] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 02:25:06,175 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-60) [495ccfc2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:25:06,178 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-60) [14967fe1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:25:06,179 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-60) [14967fe1] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 02:25:06,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [68097071] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 779280d1 >2013-03-11 02:25:06,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [68097071] FINISH, GetHardwareInfoVDSCommand, log id: 779280d1 >2013-03-11 02:25:06,256 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-12) [5489f8fd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:25:06,258 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-12) [5d232afa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:25:06,260 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-12) [5d232afa] FINISH, ActivateVdsVDSCommand, log id: 1fc92bae >2013-03-11 02:25:06,279 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-12) [5d232afa] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:25:06,280 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-12) [5d232afa] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 02:25:06,281 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [5d232afa] Autorecovering 1 storage domains >2013-03-11 02:25:06,281 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [5d232afa] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 02:25:06,282 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-12) [2a001dd4] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:25:06,282 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-12) [2a001dd4] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:25 AM >2013-03-11 02:25:06,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-12) [2a001dd4] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:25 AM >2013-03-11 02:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [525b6a6c] Autorecovering 1 hosts >2013-03-11 02:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [525b6a6c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [11e17215] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [11e17215] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [11e17215] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 197d543e >2013-03-11 02:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [11e17215] FINISH, SetVdsStatusVDSCommand, log id: 197d543e >2013-03-11 02:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-22) [11e17215] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3c022705 >2013-03-11 02:30:00,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [11e17215] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1170bcf2 >2013-03-11 02:30:00,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [11e17215] FINISH, GetHardwareInfoVDSCommand, log id: 1170bcf2 >2013-03-11 02:30:00,096 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [1c6944ad] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:30:00,098 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [5388ddca] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:30:00,099 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-22) [5388ddca] FINISH, ActivateVdsVDSCommand, log id: 3c022705 >2013-03-11 02:30:00,118 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [5388ddca] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:30:00,119 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [5388ddca] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 02:30:00,121 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [5388ddca] Autorecovering 1 storage domains >2013-03-11 02:30:00,121 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [5388ddca] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 02:30:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [19f930fd] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:30:00,122 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [19f930fd] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:30 AM >2013-03-11 02:30:00,124 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [19f930fd] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:30 AM >2013-03-11 02:30:00,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [7b2426a9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 449269e5 >2013-03-11 02:30:00,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [7b2426a9] FINISH, GetHardwareInfoVDSCommand, log id: 449269e5 >2013-03-11 02:30:00,465 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-53) [34cae325] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 02:30:00,469 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-53) [454aa06b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 02:30:00,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 42b003cf >2013-03-11 02:30:08,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 02:30:08,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:30:08,517 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 02:30:08,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 42b003cf >2013-03-11 02:30:08,521 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [44f2a259] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:30:08,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [44f2a259] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6e14005c >2013-03-11 02:30:08,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [44f2a259] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 02:30:08,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [44f2a259] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6e14005c >2013-03-11 02:30:08,539 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [454aa06b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 02:30:08,549 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-53) [6d0df775] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:30:08,550 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [6d0df775] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 723992cf >2013-03-11 02:30:08,551 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [6d0df775] FINISH, SetVdsStatusVDSCommand, log id: 723992cf >2013-03-11 02:30:08,556 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-53) [6d0df775] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 02:30:08,577 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-53) [78a9795b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:30:08,580 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-53) [1edc5e4b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:30:08,581 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-53) [1edc5e4b] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 02:30:33,867 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-52) [719e370d] Start refreshing all users data >2013-03-11 02:30:35,044 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-69) [35c5f85c] Attempting to update VMs/Templates Ovf. >2013-03-11 02:30:35,045 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-69) [35c5f85c] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 02:30:35,046 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-69) [35c5f85c] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 02:30:35,046 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-69) [35c5f85c] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 02:30:35,047 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-69) [35c5f85c] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 02:30:35,047 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-69) [35c5f85c] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 02:30:35,047 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-69) [35c5f85c] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 02:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [11c2ff1d] Autorecovering 1 hosts >2013-03-11 02:35:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [11c2ff1d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:35:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [2ca2c13e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [2ca2c13e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [2ca2c13e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6fe2ba57 >2013-03-11 02:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [2ca2c13e] FINISH, SetVdsStatusVDSCommand, log id: 6fe2ba57 >2013-03-11 02:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [2ca2c13e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 75d96e1a >2013-03-11 02:35:00,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [2ca2c13e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 476e384e >2013-03-11 02:35:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [2ca2c13e] FINISH, GetHardwareInfoVDSCommand, log id: 476e384e >2013-03-11 02:35:00,183 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [3450e9f4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:35:00,185 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [34ce8950] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:35:00,187 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [34ce8950] FINISH, ActivateVdsVDSCommand, log id: 75d96e1a >2013-03-11 02:35:00,206 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [34ce8950] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:35:00,207 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [34ce8950] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 02:35:00,208 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [34ce8950] Autorecovering 1 storage domains >2013-03-11 02:35:00,208 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [34ce8950] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 02:35:00,209 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [1bc1025e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:35:00,209 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [1bc1025e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:35 AM >2013-03-11 02:35:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [1bc1025e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:35 AM >2013-03-11 02:35:02,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [491d7e41] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 29a6bdf >2013-03-11 02:35:02,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [491d7e41] FINISH, GetHardwareInfoVDSCommand, log id: 29a6bdf >2013-03-11 02:35:02,861 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-8) [66ff67e7] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 02:35:02,865 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-8) [409f9b13] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 02:35:02,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 329c894c >2013-03-11 02:35:09,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 02:35:09,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:35:09,855 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 02:35:09,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 329c894c >2013-03-11 02:35:09,858 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [1b5eeafe] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:35:09,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [1b5eeafe] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 696cc98c >2013-03-11 02:35:09,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [1b5eeafe] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 02:35:09,878 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [1b5eeafe] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 696cc98c >2013-03-11 02:35:09,880 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [1b5eeafe] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 02:35:09,880 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [409f9b13] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 02:35:09,890 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-8) [5d6cf115] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:35:09,891 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [5d6cf115] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 75b463ac >2013-03-11 02:35:09,893 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [5d6cf115] FINISH, SetVdsStatusVDSCommand, log id: 75b463ac >2013-03-11 02:35:09,899 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-8) [5d6cf115] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 02:35:09,921 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [44569a04] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:35:09,924 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [148277f0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:35:09,925 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-8) [148277f0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 02:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [11fce17e] Autorecovering 1 hosts >2013-03-11 02:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [11fce17e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [739c66c1] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [739c66c1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [739c66c1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 702005d7 >2013-03-11 02:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [739c66c1] FINISH, SetVdsStatusVDSCommand, log id: 702005d7 >2013-03-11 02:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-75) [739c66c1] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 27b25665 >2013-03-11 02:40:00,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [739c66c1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 37d60c4 >2013-03-11 02:40:00,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [739c66c1] FINISH, GetHardwareInfoVDSCommand, log id: 37d60c4 >2013-03-11 02:40:00,412 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [176950a6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:40:00,415 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [31ddb108] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:40:00,416 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-75) [31ddb108] FINISH, ActivateVdsVDSCommand, log id: 27b25665 >2013-03-11 02:40:00,436 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [31ddb108] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:40:00,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-75) [31ddb108] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 02:40:00,438 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [31ddb108] Autorecovering 1 storage domains >2013-03-11 02:40:00,438 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [31ddb108] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 02:40:00,439 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [52abe5b9] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:40:00,439 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [52abe5b9] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:40 AM >2013-03-11 02:40:00,444 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [52abe5b9] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:40 AM >2013-03-11 02:40:01,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [497893c7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 66527258 >2013-03-11 02:40:01,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [497893c7] FINISH, GetHardwareInfoVDSCommand, log id: 66527258 >2013-03-11 02:40:01,210 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-39) [3eb2c58d] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 02:40:01,214 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-39) [5b2e93bb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 02:40:01,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1b5a73a8 >2013-03-11 02:40:10,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 02:40:10,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:40:10,214 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 02:40:10,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 1b5a73a8 >2013-03-11 02:40:10,218 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [5d50e043] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:40:10,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [5d50e043] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 33ae5c4f >2013-03-11 02:40:10,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [5d50e043] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 02:40:10,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [5d50e043] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 33ae5c4f >2013-03-11 02:40:10,233 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [5d50e043] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 02:40:10,234 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [5b2e93bb] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 02:40:10,243 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-39) [7ed3b2cb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:40:10,245 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [7ed3b2cb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3bf2a172 >2013-03-11 02:40:10,246 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [7ed3b2cb] FINISH, SetVdsStatusVDSCommand, log id: 3bf2a172 >2013-03-11 02:40:10,262 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-39) [7ed3b2cb] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 02:40:10,285 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-39) [4d91273c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:40:10,288 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-39) [7ad21c25] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:40:10,290 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-39) [7ad21c25] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 02:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [19f930fd] Autorecovering 1 hosts >2013-03-11 02:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [19f930fd] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [2eddeed7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [2eddeed7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [2eddeed7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: ff4c23 >2013-03-11 02:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [2eddeed7] FINISH, SetVdsStatusVDSCommand, log id: ff4c23 >2013-03-11 02:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-22) [2eddeed7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 41b7dddd >2013-03-11 02:45:00,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [2eddeed7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2c3ef35e >2013-03-11 02:45:00,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [2eddeed7] FINISH, GetHardwareInfoVDSCommand, log id: 2c3ef35e >2013-03-11 02:45:00,256 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [fe53c5a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:45:00,258 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [13088c8d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:45:00,259 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-22) [13088c8d] FINISH, ActivateVdsVDSCommand, log id: 41b7dddd >2013-03-11 02:45:00,278 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [13088c8d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:45:00,279 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [13088c8d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 02:45:00,280 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [13088c8d] Autorecovering 1 storage domains >2013-03-11 02:45:00,280 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [13088c8d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 02:45:00,281 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [4c69db44] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:45:00,281 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [4c69db44] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:45 AM >2013-03-11 02:45:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [4c69db44] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:45 AM >2013-03-11 02:45:01,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [58b23b68] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 35ee3b33 >2013-03-11 02:45:01,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [58b23b68] FINISH, GetHardwareInfoVDSCommand, log id: 35ee3b33 >2013-03-11 02:45:01,561 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-41) [7416d3e9] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 02:45:01,564 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-41) [dabf762] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 02:45:01,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 625146eb >2013-03-11 02:45:07,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 02:45:07,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:45:07,560 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 02:45:07,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 625146eb >2013-03-11 02:45:07,564 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [15118cee] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:45:07,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [15118cee] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: d195963 >2013-03-11 02:45:07,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [15118cee] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 02:45:07,578 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [15118cee] FINISH, MarkPoolInReconstructModeVDSCommand, log id: d195963 >2013-03-11 02:45:07,580 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [15118cee] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 02:45:07,580 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [dabf762] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 02:45:07,593 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-41) [18735480] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:45:07,594 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [18735480] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6cab8eea >2013-03-11 02:45:07,595 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [18735480] FINISH, SetVdsStatusVDSCommand, log id: 6cab8eea >2013-03-11 02:45:07,601 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-41) [18735480] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 02:45:07,624 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-41) [4db4f785] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:45:07,626 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-41) [1c0887d6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:45:07,628 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-41) [1c0887d6] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 02:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [6a7362e8] Autorecovering 1 hosts >2013-03-11 02:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [6a7362e8] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-100) [6d3c68e2] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-100) [6d3c68e2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [6d3c68e2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2aed8cbe >2013-03-11 02:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-100) [6d3c68e2] FINISH, SetVdsStatusVDSCommand, log id: 2aed8cbe >2013-03-11 02:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-100) [6d3c68e2] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7602055a >2013-03-11 02:50:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [6d3c68e2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6da3e3fb >2013-03-11 02:50:00,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-100) [6d3c68e2] FINISH, GetHardwareInfoVDSCommand, log id: 6da3e3fb >2013-03-11 02:50:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-100) [34f9271e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:50:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-100) [196a3b5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:50:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-100) [196a3b5] FINISH, ActivateVdsVDSCommand, log id: 7602055a >2013-03-11 02:50:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-100) [196a3b5] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:50:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-100) [196a3b5] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 02:50:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [196a3b5] Autorecovering 1 storage domains >2013-03-11 02:50:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-100) [196a3b5] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 02:50:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-100) [3f8f1981] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:50:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-100) [3f8f1981] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:50 AM >2013-03-11 02:50:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-100) [3f8f1981] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:50 AM >2013-03-11 02:50:01,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [7fb66044] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 595c52ed >2013-03-11 02:50:01,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [7fb66044] FINISH, GetHardwareInfoVDSCommand, log id: 595c52ed >2013-03-11 02:50:01,904 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-47) [3aa2fa00] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 02:50:01,907 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-47) [552c547] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 02:50:01,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2ba571dd >2013-03-11 02:50:07,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 02:50:07,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:50:07,884 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 02:50:07,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 2ba571dd >2013-03-11 02:50:07,888 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [63d665b3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:50:07,901 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [63d665b3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 666d3cd7 >2013-03-11 02:50:07,901 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [63d665b3] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 02:50:07,901 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [63d665b3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 666d3cd7 >2013-03-11 02:50:07,904 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [63d665b3] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 02:50:07,904 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [552c547] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 02:50:07,914 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-47) [13ca75a9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:50:07,915 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [13ca75a9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 42901cd3 >2013-03-11 02:50:07,917 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [13ca75a9] FINISH, SetVdsStatusVDSCommand, log id: 42901cd3 >2013-03-11 02:50:07,923 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [13ca75a9] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 02:50:07,957 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [2a2225b0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:50:07,959 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [59d2b06c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:50:07,961 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-47) [59d2b06c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 02:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [26f6c0fe] Autorecovering 1 hosts >2013-03-11 02:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [26f6c0fe] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [7e217ade] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [7e217ade] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [7e217ade] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 20514c0c >2013-03-11 02:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [7e217ade] FINISH, SetVdsStatusVDSCommand, log id: 20514c0c >2013-03-11 02:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [7e217ade] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 89d094b >2013-03-11 02:55:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [7e217ade] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 61df92d2 >2013-03-11 02:55:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [7e217ade] FINISH, GetHardwareInfoVDSCommand, log id: 61df92d2 >2013-03-11 02:55:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-42) [522da81e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:55:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-42) [3f55b8ee] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:55:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [3f55b8ee] FINISH, ActivateVdsVDSCommand, log id: 89d094b >2013-03-11 02:55:00,221 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [3f55b8ee] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 02:55:00,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-42) [3f55b8ee] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 02:55:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [3f55b8ee] Autorecovering 1 storage domains >2013-03-11 02:55:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [3f55b8ee] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 02:55:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [6cc0757f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:55:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [6cc0757f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:55 AM >2013-03-11 02:55:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [6cc0757f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:55 AM >2013-03-11 02:55:02,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [52abe5b9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2684d98e >2013-03-11 02:55:02,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [52abe5b9] FINISH, GetHardwareInfoVDSCommand, log id: 2684d98e >2013-03-11 02:55:02,245 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-75) [469bf42] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 02:55:02,249 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-75) [287fa06e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 02:55:02,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: bbc5678 >2013-03-11 02:55:09,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 02:55:09,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 02:55:09,232 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 02:55:09,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: bbc5678 >2013-03-11 02:55:09,236 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [2115249e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 02:55:09,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [2115249e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 367a2ceb >2013-03-11 02:55:09,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [2115249e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 02:55:09,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [2115249e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 367a2ceb >2013-03-11 02:55:09,254 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [2115249e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 02:55:09,254 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [287fa06e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 02:55:09,265 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-75) [71933ce2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:55:09,266 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [71933ce2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6bff92ea >2013-03-11 02:55:09,267 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [71933ce2] FINISH, SetVdsStatusVDSCommand, log id: 6bff92ea >2013-03-11 02:55:09,273 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-75) [71933ce2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 02:55:09,296 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [1b9522c8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:55:09,298 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [7199e66f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 02:55:09,300 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-75) [7199e66f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 03:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [8051887] Autorecovering 1 hosts >2013-03-11 03:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [8051887] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [77d21445] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [77d21445] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [77d21445] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 65655210 >2013-03-11 03:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [77d21445] FINISH, SetVdsStatusVDSCommand, log id: 65655210 >2013-03-11 03:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-33) [77d21445] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 40cb74 >2013-03-11 03:00:00,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [77d21445] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 9bb28c7 >2013-03-11 03:00:00,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [77d21445] FINISH, GetHardwareInfoVDSCommand, log id: 9bb28c7 >2013-03-11 03:00:00,650 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-33) [6a490a37] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:00:00,653 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-33) [76cb487a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:00:00,654 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-33) [76cb487a] FINISH, ActivateVdsVDSCommand, log id: 40cb74 >2013-03-11 03:00:00,674 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [76cb487a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:00:00,676 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-33) [76cb487a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 03:00:00,677 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [76cb487a] Autorecovering 1 storage domains >2013-03-11 03:00:00,678 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [76cb487a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 03:00:00,678 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [7c928d83] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:00:00,678 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [7c928d83] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:00 AM >2013-03-11 03:00:00,681 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [7c928d83] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:00 AM >2013-03-11 03:00:00,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [5664a7d8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2a723c6a >2013-03-11 03:00:00,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [5664a7d8] FINISH, GetHardwareInfoVDSCommand, log id: 2a723c6a >2013-03-11 03:00:00,762 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-15) [63c94dee] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 03:00:00,765 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-15) [5bee11c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 03:00:00,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: cd2231a >2013-03-11 03:00:06,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 03:00:06,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:00:06,824 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 03:00:06,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: cd2231a >2013-03-11 03:00:06,828 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [6eba5f1c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:00:06,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [6eba5f1c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1a0b89b3 >2013-03-11 03:00:06,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [6eba5f1c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 03:00:06,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [6eba5f1c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1a0b89b3 >2013-03-11 03:00:06,843 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [6eba5f1c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 03:00:06,844 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [5bee11c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 03:00:06,853 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-15) [5eeb7172] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:00:06,854 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [5eeb7172] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 305ea7d4 >2013-03-11 03:00:06,856 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [5eeb7172] FINISH, SetVdsStatusVDSCommand, log id: 305ea7d4 >2013-03-11 03:00:06,861 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-15) [5eeb7172] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 03:00:06,884 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-15) [35400a45] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:00:06,886 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-15) [6b6d9112] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:00:06,887 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-15) [6b6d9112] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 03:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) [53ecd05b] Autorecovering 1 hosts >2013-03-11 03:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) [53ecd05b] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-61) [3e8e2f57] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-61) [3e8e2f57] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [3e8e2f57] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7fa44dfd >2013-03-11 03:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [3e8e2f57] FINISH, SetVdsStatusVDSCommand, log id: 7fa44dfd >2013-03-11 03:05:00,010 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-61) [3e8e2f57] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2188a142 >2013-03-11 03:05:00,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [3e8e2f57] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 56007be9 >2013-03-11 03:05:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [3e8e2f57] FINISH, GetHardwareInfoVDSCommand, log id: 56007be9 >2013-03-11 03:05:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-61) [65779c01] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:05:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-61) [3c3cc451] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:05:00,093 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-61) [3c3cc451] FINISH, ActivateVdsVDSCommand, log id: 2188a142 >2013-03-11 03:05:00,112 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-61) [3c3cc451] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:05:00,113 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-61) [3c3cc451] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 03:05:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) [3c3cc451] Autorecovering 1 storage domains >2013-03-11 03:05:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) [3c3cc451] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 03:05:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-61) [3d0b7d2d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:05:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-61) [3d0b7d2d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:05 AM >2013-03-11 03:05:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-61) [3d0b7d2d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:05 AM >2013-03-11 03:05:01,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [4dc160a6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 35c40da7 >2013-03-11 03:05:01,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [4dc160a6] FINISH, GetHardwareInfoVDSCommand, log id: 35c40da7 >2013-03-11 03:05:01,166 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-97) [146722e9] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 03:05:01,170 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-97) [52a9dcd6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 03:05:01,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7e70d685 >2013-03-11 03:05:08,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 03:05:08,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:05:08,187 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 03:05:08,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 7e70d685 >2013-03-11 03:05:08,191 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [26363c72] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:05:08,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [26363c72] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4cdac5d0 >2013-03-11 03:05:08,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [26363c72] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 03:05:08,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [26363c72] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4cdac5d0 >2013-03-11 03:05:08,206 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [26363c72] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 03:05:08,207 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [52a9dcd6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 03:05:08,216 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-97) [517247cc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:05:08,217 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [517247cc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 77159dfa >2013-03-11 03:05:08,219 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [517247cc] FINISH, SetVdsStatusVDSCommand, log id: 77159dfa >2013-03-11 03:05:08,236 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-97) [517247cc] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 03:05:08,259 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-97) [3db7578d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:05:08,262 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-97) [25427d38] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:05:08,263 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-97) [25427d38] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 03:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) [35ad7cab] Autorecovering 1 hosts >2013-03-11 03:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) [35ad7cab] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-11) [730f4541] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-11) [730f4541] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [730f4541] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 12b086e8 >2013-03-11 03:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [730f4541] FINISH, SetVdsStatusVDSCommand, log id: 12b086e8 >2013-03-11 03:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-11) [730f4541] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 659f9fbf >2013-03-11 03:10:00,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [730f4541] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 35182c6 >2013-03-11 03:10:00,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [730f4541] FINISH, GetHardwareInfoVDSCommand, log id: 35182c6 >2013-03-11 03:10:00,206 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-11) [1abc435d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:10:00,209 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-11) [3352121d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:10:00,210 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-11) [3352121d] FINISH, ActivateVdsVDSCommand, log id: 659f9fbf >2013-03-11 03:10:00,229 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-11) [3352121d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:10:00,230 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-11) [3352121d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 03:10:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) [3352121d] Autorecovering 1 storage domains >2013-03-11 03:10:00,232 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) [3352121d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 03:10:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-11) [7d18a3ff] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:10:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-11) [7d18a3ff] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:10 AM >2013-03-11 03:10:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-11) [7d18a3ff] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:10 AM >2013-03-11 03:10:02,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [113b057b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5eb71bb2 >2013-03-11 03:10:02,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [113b057b] FINISH, GetHardwareInfoVDSCommand, log id: 5eb71bb2 >2013-03-11 03:10:02,576 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-96) [527b5be1] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 03:10:02,579 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-96) [3907aa0d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 03:10:02,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1c20cd55 >2013-03-11 03:10:08,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 03:10:08,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:10:08,627 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 03:10:08,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 1c20cd55 >2013-03-11 03:10:08,631 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [59a502ae] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:10:08,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [59a502ae] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: b4763b7 >2013-03-11 03:10:08,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [59a502ae] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 03:10:08,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [59a502ae] FINISH, MarkPoolInReconstructModeVDSCommand, log id: b4763b7 >2013-03-11 03:10:08,647 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [59a502ae] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 03:10:08,648 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [3907aa0d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 03:10:08,657 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-96) [3bbd6c27] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:10:08,659 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [3bbd6c27] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 734ee8c4 >2013-03-11 03:10:08,660 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [3bbd6c27] FINISH, SetVdsStatusVDSCommand, log id: 734ee8c4 >2013-03-11 03:10:08,666 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-96) [3bbd6c27] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 03:10:08,701 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-96) [bf4a81a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:10:08,703 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-96) [521f7032] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:10:08,704 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-96) [521f7032] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 03:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [41b549dc] Autorecovering 1 hosts >2013-03-11 03:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [41b549dc] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [be7caf] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [be7caf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [be7caf] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6f4d6cfe >2013-03-11 03:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [be7caf] FINISH, SetVdsStatusVDSCommand, log id: 6f4d6cfe >2013-03-11 03:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [be7caf] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7d3ec590 >2013-03-11 03:15:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [be7caf] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2cde24c2 >2013-03-11 03:15:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [be7caf] FINISH, GetHardwareInfoVDSCommand, log id: 2cde24c2 >2013-03-11 03:15:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [7b7340e0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:15:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [54ab5dfb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:15:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [54ab5dfb] FINISH, ActivateVdsVDSCommand, log id: 7d3ec590 >2013-03-11 03:15:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [54ab5dfb] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:15:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [54ab5dfb] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 03:15:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [54ab5dfb] Autorecovering 1 storage domains >2013-03-11 03:15:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [54ab5dfb] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 03:15:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [5e22dc1b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:15:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [5e22dc1b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:15 AM >2013-03-11 03:15:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [5e22dc1b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:15 AM >2013-03-11 03:15:02,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [27d84c31] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 10e92f0c >2013-03-11 03:15:02,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [27d84c31] FINISH, GetHardwareInfoVDSCommand, log id: 10e92f0c >2013-03-11 03:15:02,986 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-81) [33ae30cc] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 03:15:02,989 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-81) [749308b8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 03:15:02,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: acb180c >2013-03-11 03:15:09,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 03:15:09,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:15:09,993 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 03:15:09,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: acb180c >2013-03-11 03:15:09,997 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [89190ff] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:15:10,010 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [89190ff] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 247c5daf >2013-03-11 03:15:10,010 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [89190ff] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 03:15:10,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [89190ff] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 247c5daf >2013-03-11 03:15:10,013 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [89190ff] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 03:15:10,013 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [749308b8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 03:15:10,022 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-81) [3c152c82] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:15:10,024 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [3c152c82] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 473d7fd7 >2013-03-11 03:15:10,025 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [3c152c82] FINISH, SetVdsStatusVDSCommand, log id: 473d7fd7 >2013-03-11 03:15:10,035 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-81) [3c152c82] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 03:15:10,057 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-81) [3786799c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:15:10,060 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-81) [1dc1a745] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:15:10,061 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-81) [1dc1a745] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 03:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [6335001a] Autorecovering 1 hosts >2013-03-11 03:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [6335001a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [64a9879f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [64a9879f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [64a9879f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3f89b433 >2013-03-11 03:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [64a9879f] FINISH, SetVdsStatusVDSCommand, log id: 3f89b433 >2013-03-11 03:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-1) [64a9879f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7c66e172 >2013-03-11 03:20:00,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [64a9879f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6d72b021 >2013-03-11 03:20:00,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [64a9879f] FINISH, GetHardwareInfoVDSCommand, log id: 6d72b021 >2013-03-11 03:20:00,259 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-1) [7019e4ec] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:20:00,262 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-1) [77ba7929] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:20:00,263 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-1) [77ba7929] FINISH, ActivateVdsVDSCommand, log id: 7c66e172 >2013-03-11 03:20:00,282 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [77ba7929] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:20:00,283 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-1) [77ba7929] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 03:20:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [77ba7929] Autorecovering 1 storage domains >2013-03-11 03:20:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [77ba7929] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 03:20:00,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [61dbcbeb] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:20:00,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [61dbcbeb] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:20 AM >2013-03-11 03:20:00,288 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [61dbcbeb] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:20 AM >2013-03-11 03:20:01,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [209f3120] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 79c1d409 >2013-03-11 03:20:01,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [209f3120] FINISH, GetHardwareInfoVDSCommand, log id: 79c1d409 >2013-03-11 03:20:01,335 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-92) [508bb725] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 03:20:01,339 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-92) [4f5fb6f0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 03:20:01,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1bedd324 >2013-03-11 03:20:08,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 03:20:08,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:20:08,337 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 03:20:08,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 1bedd324 >2013-03-11 03:20:08,340 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [33d0c026] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:20:08,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [33d0c026] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1163148 >2013-03-11 03:20:08,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [33d0c026] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 03:20:08,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [33d0c026] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1163148 >2013-03-11 03:20:08,356 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [33d0c026] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 03:20:08,357 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [4f5fb6f0] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 03:20:08,366 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-92) [60f97ec6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:20:08,367 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [60f97ec6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4aecd6de >2013-03-11 03:20:08,369 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [60f97ec6] FINISH, SetVdsStatusVDSCommand, log id: 4aecd6de >2013-03-11 03:20:08,374 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-92) [60f97ec6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 03:20:08,398 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-92) [3e0d23f0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:20:08,401 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-92) [406be437] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:20:08,402 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-92) [406be437] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 03:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [7a5aeb22] Autorecovering 1 hosts >2013-03-11 03:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [7a5aeb22] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [1017c458] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [1017c458] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [1017c458] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5ea4cb1a >2013-03-11 03:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [1017c458] FINISH, SetVdsStatusVDSCommand, log id: 5ea4cb1a >2013-03-11 03:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-19) [1017c458] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4b4ae952 >2013-03-11 03:25:00,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [1017c458] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 192fc024 >2013-03-11 03:25:00,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [1017c458] FINISH, GetHardwareInfoVDSCommand, log id: 192fc024 >2013-03-11 03:25:00,413 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-19) [5af1f2cc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:25:00,416 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-19) [35cc9e70] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:25:00,417 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-19) [35cc9e70] FINISH, ActivateVdsVDSCommand, log id: 4b4ae952 >2013-03-11 03:25:00,436 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [35cc9e70] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:25:00,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-19) [35cc9e70] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 03:25:00,438 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [35cc9e70] Autorecovering 1 storage domains >2013-03-11 03:25:00,438 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [35cc9e70] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 03:25:00,439 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [38e7314c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:25:00,439 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [38e7314c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:25 AM >2013-03-11 03:25:00,442 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [38e7314c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:25 AM >2013-03-11 03:25:02,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [2a634207] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 65db8a21 >2013-03-11 03:25:02,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [2a634207] FINISH, GetHardwareInfoVDSCommand, log id: 65db8a21 >2013-03-11 03:25:02,677 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-54) [6e2326b1] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 03:25:02,680 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-54) [49d2fd03] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 03:25:02,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 461c453b >2013-03-11 03:25:07,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 03:25:07,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:25:07,658 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 03:25:07,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 461c453b >2013-03-11 03:25:07,661 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [34ad4d5a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:25:07,679 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [34ad4d5a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c40e653 >2013-03-11 03:25:07,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [34ad4d5a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 03:25:07,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [34ad4d5a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c40e653 >2013-03-11 03:25:07,685 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [34ad4d5a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 03:25:07,685 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [49d2fd03] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 03:25:07,696 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-54) [7863a1a1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:25:07,697 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [7863a1a1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4ae292fd >2013-03-11 03:25:07,698 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [7863a1a1] FINISH, SetVdsStatusVDSCommand, log id: 4ae292fd >2013-03-11 03:25:07,704 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [7863a1a1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 03:25:07,728 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [4efd06fe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:25:07,731 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [1ea9775] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:25:07,733 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-54) [1ea9775] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 03:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) [5f78ab61] Autorecovering 1 hosts >2013-03-11 03:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) [5f78ab61] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-85) [339b53] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-85) [339b53] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-85) [339b53] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 15d23b26 >2013-03-11 03:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-85) [339b53] FINISH, SetVdsStatusVDSCommand, log id: 15d23b26 >2013-03-11 03:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-85) [339b53] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1300989f >2013-03-11 03:30:00,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-85) [339b53] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 42d1b6c4 >2013-03-11 03:30:00,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-85) [339b53] FINISH, GetHardwareInfoVDSCommand, log id: 42d1b6c4 >2013-03-11 03:30:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-85) [74bb2261] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:30:00,206 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-85) [3ccb7a04] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:30:00,208 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-85) [3ccb7a04] FINISH, ActivateVdsVDSCommand, log id: 1300989f >2013-03-11 03:30:00,227 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-85) [3ccb7a04] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:30:00,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-85) [3ccb7a04] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 03:30:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) [3ccb7a04] Autorecovering 1 storage domains >2013-03-11 03:30:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) [3ccb7a04] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 03:30:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-85) [76533881] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:30:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-85) [76533881] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:30 AM >2013-03-11 03:30:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-85) [76533881] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:30 AM >2013-03-11 03:30:01,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [5c264d10] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1d9dedb8 >2013-03-11 03:30:01,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [5c264d10] FINISH, GetHardwareInfoVDSCommand, log id: 1d9dedb8 >2013-03-11 03:30:02,016 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-36) [3b53a410] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 03:30:02,019 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-36) [1b9a9f76] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 03:30:02,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 29e436e1 >2013-03-11 03:30:11,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 03:30:11,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:30:11,071 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 03:30:11,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 29e436e1 >2013-03-11 03:30:11,074 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [7e2924aa] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:30:11,087 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [7e2924aa] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 33d54b9 >2013-03-11 03:30:11,087 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [7e2924aa] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 03:30:11,087 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [7e2924aa] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 33d54b9 >2013-03-11 03:30:11,092 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [1b9a9f76] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 03:30:11,102 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-36) [541dfce4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:30:11,104 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [541dfce4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6a88ff35 >2013-03-11 03:30:11,105 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [541dfce4] FINISH, SetVdsStatusVDSCommand, log id: 6a88ff35 >2013-03-11 03:30:11,111 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [541dfce4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 03:30:11,137 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [59081214] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:30:11,139 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [524c4662] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:30:11,140 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-36) [524c4662] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 03:30:33,868 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-35) [3c6eb9e5] Start refreshing all users data >2013-03-11 03:30:35,047 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [6dc1a565] Attempting to update VMs/Templates Ovf. >2013-03-11 03:30:35,049 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [6dc1a565] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 03:30:35,049 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [6dc1a565] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 03:30:35,050 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [6dc1a565] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 03:30:35,050 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [6dc1a565] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 03:30:35,050 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [6dc1a565] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 03:30:35,051 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-5) [6dc1a565] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 03:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [61dbcbeb] Autorecovering 1 hosts >2013-03-11 03:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [61dbcbeb] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [d5c988d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [d5c988d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [d5c988d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: b3a9602 >2013-03-11 03:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [d5c988d] FINISH, SetVdsStatusVDSCommand, log id: b3a9602 >2013-03-11 03:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-1) [d5c988d] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5f06b3c0 >2013-03-11 03:35:00,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [d5c988d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 56e61a0b >2013-03-11 03:35:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [d5c988d] FINISH, GetHardwareInfoVDSCommand, log id: 56e61a0b >2013-03-11 03:35:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-1) [3810d1c6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:35:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-1) [14fc68de] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:35:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-1) [14fc68de] FINISH, ActivateVdsVDSCommand, log id: 5f06b3c0 >2013-03-11 03:35:00,219 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [14fc68de] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:35:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-1) [14fc68de] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 03:35:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [14fc68de] Autorecovering 1 storage domains >2013-03-11 03:35:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [14fc68de] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 03:35:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [7f66581b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:35:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [7f66581b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:35 AM >2013-03-11 03:35:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [7f66581b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:35 AM >2013-03-11 03:35:02,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [3d0b7d2d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 306b2615 >2013-03-11 03:35:02,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [3d0b7d2d] FINISH, GetHardwareInfoVDSCommand, log id: 306b2615 >2013-03-11 03:35:02,428 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-61) [1dee5845] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 03:35:02,431 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-61) [464d8691] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 03:35:02,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 20416637 >2013-03-11 03:35:10,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 03:35:10,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:35:10,428 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 03:35:10,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 20416637 >2013-03-11 03:35:10,432 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [17d6c45c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:35:10,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [17d6c45c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 54bf163c >2013-03-11 03:35:10,446 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [17d6c45c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 03:35:10,446 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [17d6c45c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 54bf163c >2013-03-11 03:35:10,448 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [17d6c45c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 03:35:10,448 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [464d8691] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 03:35:10,457 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-61) [3db5f2ea] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:35:10,459 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [3db5f2ea] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: bdd21b4 >2013-03-11 03:35:10,460 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [3db5f2ea] FINISH, SetVdsStatusVDSCommand, log id: bdd21b4 >2013-03-11 03:35:10,466 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-61) [3db5f2ea] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 03:35:10,492 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-61) [8c2f3f7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:35:10,494 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-61) [56ead237] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:35:10,496 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-61) [56ead237] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (QuartzScheduler_Worker-36) [524c4662] AuditLogCleanupManager::deleteAgedOutAuditLogs - entered >2013-03-11 03:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [1849eaac] Autorecovering 1 hosts >2013-03-11 03:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [1849eaac] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [44bef54d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [44bef54d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [44bef54d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 418861c5 >2013-03-11 03:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [44bef54d] FINISH, SetVdsStatusVDSCommand, log id: 418861c5 >2013-03-11 03:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-93) [44bef54d] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 764be489 >2013-03-11 03:40:00,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [44bef54d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 16d356ca >2013-03-11 03:40:00,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [44bef54d] FINISH, GetHardwareInfoVDSCommand, log id: 16d356ca >2013-03-11 03:40:00,187 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [240a3685] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:40:00,190 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [3b23d258] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:40:00,191 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-93) [3b23d258] FINISH, ActivateVdsVDSCommand, log id: 764be489 >2013-03-11 03:40:00,210 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [3b23d258] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:40:00,211 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-93) [3b23d258] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 03:40:00,212 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [3b23d258] Autorecovering 1 storage domains >2013-03-11 03:40:00,212 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [3b23d258] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 03:40:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [29b43c6b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:40:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [29b43c6b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:40 AM >2013-03-11 03:40:00,216 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [29b43c6b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:40 AM >2013-03-11 03:40:01,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [38e7314c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: a6f3c6e >2013-03-11 03:40:01,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [38e7314c] FINISH, GetHardwareInfoVDSCommand, log id: a6f3c6e >2013-03-11 03:40:01,780 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-19) [12660ccf] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 03:40:01,784 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-19) [3447ed3d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 03:40:01,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 364a9092 >2013-03-11 03:40:08,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 03:40:08,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:40:08,799 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 03:40:08,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 364a9092 >2013-03-11 03:40:08,802 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [7a28ec33] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:40:08,816 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [7a28ec33] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 435eea1f >2013-03-11 03:40:08,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [7a28ec33] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 03:40:08,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [7a28ec33] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 435eea1f >2013-03-11 03:40:08,819 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [7a28ec33] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 03:40:08,819 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [3447ed3d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 03:40:08,828 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-19) [4f224940] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:40:08,829 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [4f224940] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2ea2e1b >2013-03-11 03:40:08,831 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [4f224940] FINISH, SetVdsStatusVDSCommand, log id: 2ea2e1b >2013-03-11 03:40:08,837 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-19) [4f224940] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 03:40:08,870 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-19) [517d0f79] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:40:08,873 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-19) [32116524] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:40:08,874 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-19) [32116524] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 03:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) [76533881] Autorecovering 1 hosts >2013-03-11 03:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) [76533881] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-85) [62427e75] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-85) [62427e75] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-85) [62427e75] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2ae6446c >2013-03-11 03:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-85) [62427e75] FINISH, SetVdsStatusVDSCommand, log id: 2ae6446c >2013-03-11 03:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-85) [62427e75] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3c57d15 >2013-03-11 03:45:00,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-85) [62427e75] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6cbee7a0 >2013-03-11 03:45:00,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-85) [62427e75] FINISH, GetHardwareInfoVDSCommand, log id: 6cbee7a0 >2013-03-11 03:45:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-85) [132bf2a9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:45:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-85) [58cf598d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:45:00,206 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-85) [58cf598d] FINISH, ActivateVdsVDSCommand, log id: 3c57d15 >2013-03-11 03:45:00,225 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-85) [58cf598d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:45:00,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-85) [58cf598d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 03:45:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) [58cf598d] Autorecovering 1 storage domains >2013-03-11 03:45:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-85) [58cf598d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 03:45:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-85) [183a8084] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:45:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-85) [183a8084] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:45 AM >2013-03-11 03:45:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-85) [183a8084] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:45 AM >2013-03-11 03:45:00,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [3268391f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2deb1e72 >2013-03-11 03:45:00,267 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [3268391f] FINISH, GetHardwareInfoVDSCommand, log id: 2deb1e72 >2013-03-11 03:45:00,303 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-24) [4f7e4b4] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 03:45:00,307 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-24) [496e5471] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 03:45:00,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: d420609 >2013-03-11 03:45:05,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 03:45:05,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:45:05,295 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 03:45:05,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: d420609 >2013-03-11 03:45:05,299 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [49b69ede] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:45:05,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [49b69ede] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 257febd3 >2013-03-11 03:45:05,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [49b69ede] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 03:45:05,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [49b69ede] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 257febd3 >2013-03-11 03:45:05,315 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [49b69ede] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 03:45:05,316 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [496e5471] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 03:45:05,325 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-24) [6f8d107] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:45:05,327 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [6f8d107] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6061802 >2013-03-11 03:45:05,328 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [6f8d107] FINISH, SetVdsStatusVDSCommand, log id: 6061802 >2013-03-11 03:45:05,334 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-24) [6f8d107] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 03:45:05,357 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-24) [3d23e8b9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:45:05,359 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-24) [137d1c0a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:45:05,361 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-24) [137d1c0a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 03:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [1c38d809] Autorecovering 1 hosts >2013-03-11 03:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [1c38d809] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-55) [1c5501ae] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-55) [1c5501ae] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [1c5501ae] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 74a6891b >2013-03-11 03:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [1c5501ae] FINISH, SetVdsStatusVDSCommand, log id: 74a6891b >2013-03-11 03:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-55) [1c5501ae] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3b0e0459 >2013-03-11 03:50:00,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [1c5501ae] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1edfe124 >2013-03-11 03:50:00,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [1c5501ae] FINISH, GetHardwareInfoVDSCommand, log id: 1edfe124 >2013-03-11 03:50:00,206 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-55) [5005694a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:50:00,209 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-55) [33c439dc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:50:00,210 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-55) [33c439dc] FINISH, ActivateVdsVDSCommand, log id: 3b0e0459 >2013-03-11 03:50:00,229 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-55) [33c439dc] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:50:00,230 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-55) [33c439dc] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 03:50:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [33c439dc] Autorecovering 1 storage domains >2013-03-11 03:50:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [33c439dc] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 03:50:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-55) [41e464c2] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:50:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-55) [41e464c2] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:50 AM >2013-03-11 03:50:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-55) [41e464c2] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:50 AM >2013-03-11 03:50:02,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [7f66581b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: b648dd0 >2013-03-11 03:50:02,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [7f66581b] FINISH, GetHardwareInfoVDSCommand, log id: b648dd0 >2013-03-11 03:50:02,650 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-1) [5c057c54] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 03:50:02,653 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-1) [5083d6e7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 03:50:02,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2ed99f83 >2013-03-11 03:50:08,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 03:50:08,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:50:08,646 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 03:50:08,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 2ed99f83 >2013-03-11 03:50:08,651 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [c384db3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:50:08,664 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [c384db3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 48adfb6e >2013-03-11 03:50:08,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [c384db3] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 03:50:08,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [c384db3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 48adfb6e >2013-03-11 03:50:08,667 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [c384db3] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 03:50:08,667 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [5083d6e7] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 03:50:08,677 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-1) [50daf525] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:50:08,678 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [50daf525] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3358b9d2 >2013-03-11 03:50:08,680 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [50daf525] FINISH, SetVdsStatusVDSCommand, log id: 3358b9d2 >2013-03-11 03:50:08,686 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-1) [50daf525] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 03:50:08,708 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-1) [5ed6a2ba] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:50:08,711 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-1) [19ab7d79] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:50:08,712 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-1) [19ab7d79] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 03:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [22d05a6a] Autorecovering 1 hosts >2013-03-11 03:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [22d05a6a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [107cfab8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [107cfab8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [107cfab8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6a2169e6 >2013-03-11 03:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [107cfab8] FINISH, SetVdsStatusVDSCommand, log id: 6a2169e6 >2013-03-11 03:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-90) [107cfab8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 546a6d91 >2013-03-11 03:55:00,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [107cfab8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 47da1567 >2013-03-11 03:55:00,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [107cfab8] FINISH, GetHardwareInfoVDSCommand, log id: 47da1567 >2013-03-11 03:55:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-90) [6993c01] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:55:00,207 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-90) [13afa4f3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:55:00,208 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-90) [13afa4f3] FINISH, ActivateVdsVDSCommand, log id: 546a6d91 >2013-03-11 03:55:00,227 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-90) [13afa4f3] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 03:55:00,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [13afa4f3] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 03:55:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [13afa4f3] Autorecovering 1 storage domains >2013-03-11 03:55:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-90) [13afa4f3] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 03:55:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [5c84216c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:55:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [5c84216c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:55 AM >2013-03-11 03:55:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-90) [5c84216c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:55 AM >2013-03-11 03:55:02,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-40) [f452ece] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 22d9d5d >2013-03-11 03:55:02,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-40) [f452ece] FINISH, GetHardwareInfoVDSCommand, log id: 22d9d5d >2013-03-11 03:55:02,988 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-40) [12edbfe9] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 03:55:02,992 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-40) [6cf92219] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 03:55:03,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5707e797 >2013-03-11 03:55:10,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 03:55:10,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 03:55:10,016 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 03:55:10,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 5707e797 >2013-03-11 03:55:10,020 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [511e74bb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 03:55:10,033 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [511e74bb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5c69a0db >2013-03-11 03:55:10,033 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [511e74bb] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 03:55:10,033 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [511e74bb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5c69a0db >2013-03-11 03:55:10,035 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [511e74bb] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 03:55:10,035 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [6cf92219] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 03:55:10,045 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-40) [242d1d04] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:55:10,046 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-40) [242d1d04] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6cb80e36 >2013-03-11 03:55:10,047 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-40) [242d1d04] FINISH, SetVdsStatusVDSCommand, log id: 6cb80e36 >2013-03-11 03:55:10,053 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-40) [242d1d04] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 03:55:10,075 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-40) [45a5da5c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:55:10,077 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-40) [779cd84] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 03:55:10,079 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-40) [779cd84] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 04:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [659d7a3f] Autorecovering 1 hosts >2013-03-11 04:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [659d7a3f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [5a5e65f5] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [5a5e65f5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [5a5e65f5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 30cb5cef >2013-03-11 04:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [5a5e65f5] FINISH, SetVdsStatusVDSCommand, log id: 30cb5cef >2013-03-11 04:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-18) [5a5e65f5] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4b99ccb3 >2013-03-11 04:00:00,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [5a5e65f5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 65bc5237 >2013-03-11 04:00:00,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [5a5e65f5] FINISH, GetHardwareInfoVDSCommand, log id: 65bc5237 >2013-03-11 04:00:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [2a08a44d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:00:00,207 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [529f0659] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:00:00,208 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-18) [529f0659] FINISH, ActivateVdsVDSCommand, log id: 4b99ccb3 >2013-03-11 04:00:00,227 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [529f0659] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:00:00,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [529f0659] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 04:00:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [529f0659] Autorecovering 1 storage domains >2013-03-11 04:00:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [529f0659] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 04:00:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [13e84359] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:00:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [13e84359] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:00 AM >2013-03-11 04:00:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [13e84359] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:00 AM >2013-03-11 04:00:01,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [2cbdecd0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5bdbd25c >2013-03-11 04:00:01,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [2cbdecd0] FINISH, GetHardwareInfoVDSCommand, log id: 5bdbd25c >2013-03-11 04:00:01,352 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-13) [3973fc44] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 04:00:01,356 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-13) [6d5be927] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 04:00:01,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2c7af639 >2013-03-11 04:00:07,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 04:00:07,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:00:07,382 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 04:00:07,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2c7af639 >2013-03-11 04:00:07,386 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [26c59925] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:00:07,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [26c59925] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 66e6ab6b >2013-03-11 04:00:07,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [26c59925] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 04:00:07,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [26c59925] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 66e6ab6b >2013-03-11 04:00:07,404 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [26c59925] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 04:00:07,404 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [6d5be927] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 04:00:07,421 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-13) [14fe3f46] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:00:07,423 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [14fe3f46] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 13fe6bad >2013-03-11 04:00:07,424 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [14fe3f46] FINISH, SetVdsStatusVDSCommand, log id: 13fe6bad >2013-03-11 04:00:07,430 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-13) [14fe3f46] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 04:00:07,453 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-13) [3eedd71c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:00:07,455 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-13) [43ef0f74] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:00:07,457 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-13) [43ef0f74] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 04:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [58333aca] Autorecovering 1 hosts >2013-03-11 04:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [58333aca] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [18ed08bb] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [18ed08bb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [18ed08bb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2dbc36b1 >2013-03-11 04:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [18ed08bb] FINISH, SetVdsStatusVDSCommand, log id: 2dbc36b1 >2013-03-11 04:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [18ed08bb] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 55ea09e7 >2013-03-11 04:05:00,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [18ed08bb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 398d5d2c >2013-03-11 04:05:00,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [18ed08bb] FINISH, GetHardwareInfoVDSCommand, log id: 398d5d2c >2013-03-11 04:05:00,264 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [33fccab5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:05:00,266 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [823c69e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:05:00,268 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [823c69e] FINISH, ActivateVdsVDSCommand, log id: 55ea09e7 >2013-03-11 04:05:00,287 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [823c69e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:05:00,288 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [823c69e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 04:05:00,289 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [823c69e] Autorecovering 1 storage domains >2013-03-11 04:05:00,289 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [823c69e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 04:05:00,290 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [80620b4] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:05:00,290 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [80620b4] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:05 AM >2013-03-11 04:05:00,293 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [80620b4] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:05 AM >2013-03-11 04:05:01,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [41e464c2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7149654f >2013-03-11 04:05:01,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [41e464c2] FINISH, GetHardwareInfoVDSCommand, log id: 7149654f >2013-03-11 04:05:01,735 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-55) [1f3ce6c2] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 04:05:01,738 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-55) [598fd1b9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 04:05:01,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4f53e5e4 >2013-03-11 04:05:08,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 04:05:08,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:05:08,748 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 04:05:08,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 4f53e5e4 >2013-03-11 04:05:08,752 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [172873ea] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:05:08,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [172873ea] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 14fe8c33 >2013-03-11 04:05:08,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [172873ea] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 04:05:08,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [172873ea] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 14fe8c33 >2013-03-11 04:05:08,767 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [172873ea] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 04:05:08,768 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [598fd1b9] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 04:05:08,778 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-55) [2badbd45] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:05:08,779 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [2badbd45] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 57586308 >2013-03-11 04:05:08,780 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [2badbd45] FINISH, SetVdsStatusVDSCommand, log id: 57586308 >2013-03-11 04:05:08,786 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-55) [2badbd45] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 04:05:08,809 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-55) [3a3ffd92] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:05:08,812 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-55) [49fe93dc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:05:08,813 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-55) [49fe93dc] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 04:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [866e613] Autorecovering 1 hosts >2013-03-11 04:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [866e613] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:10:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [52284cad] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:10:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [52284cad] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:10:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [52284cad] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5cb48b53 >2013-03-11 04:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [52284cad] FINISH, SetVdsStatusVDSCommand, log id: 5cb48b53 >2013-03-11 04:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-2) [52284cad] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7b939b22 >2013-03-11 04:10:00,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [52284cad] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 798a7bd0 >2013-03-11 04:10:00,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [52284cad] FINISH, GetHardwareInfoVDSCommand, log id: 798a7bd0 >2013-03-11 04:10:00,243 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [55a6a827] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:10:00,246 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [18f291bc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:10:00,247 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-2) [18f291bc] FINISH, ActivateVdsVDSCommand, log id: 7b939b22 >2013-03-11 04:10:00,267 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [18f291bc] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:10:00,267 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [18f291bc] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 04:10:00,269 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [18f291bc] Autorecovering 1 storage domains >2013-03-11 04:10:00,269 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [18f291bc] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 04:10:00,269 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [424759f8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:10:00,269 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [424759f8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:10 AM >2013-03-11 04:10:00,272 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [424759f8] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:10 AM >2013-03-11 04:10:03,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [54a19484] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 28139216 >2013-03-11 04:10:03,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [54a19484] FINISH, GetHardwareInfoVDSCommand, log id: 28139216 >2013-03-11 04:10:03,093 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-66) [29252809] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 04:10:03,097 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-66) [2d216996] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 04:10:03,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2e18c0c9 >2013-03-11 04:10:07,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 04:10:07,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:10:07,088 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 04:10:07,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2e18c0c9 >2013-03-11 04:10:07,092 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [73f1fe9c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:10:07,107 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [73f1fe9c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7c41097a >2013-03-11 04:10:07,107 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [73f1fe9c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 04:10:07,107 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [73f1fe9c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7c41097a >2013-03-11 04:10:07,109 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [73f1fe9c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 04:10:07,110 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [2d216996] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 04:10:07,119 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-66) [3596179e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:10:07,121 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [3596179e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 34951ac6 >2013-03-11 04:10:07,122 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [3596179e] FINISH, SetVdsStatusVDSCommand, log id: 34951ac6 >2013-03-11 04:10:07,128 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-66) [3596179e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 04:10:07,152 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-66) [4de89da8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:10:07,154 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-66) [762f7c4f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:10:07,155 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-66) [762f7c4f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 04:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-31) [70b744d8] Autorecovering 1 hosts >2013-03-11 04:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-31) [70b744d8] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-31) [14775abe] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-31) [14775abe] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [14775abe] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 51c65018 >2013-03-11 04:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [14775abe] FINISH, SetVdsStatusVDSCommand, log id: 51c65018 >2013-03-11 04:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-31) [14775abe] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5fbe4781 >2013-03-11 04:15:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [14775abe] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 374e6843 >2013-03-11 04:15:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [14775abe] FINISH, GetHardwareInfoVDSCommand, log id: 374e6843 >2013-03-11 04:15:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-31) [3c4f0f4c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:15:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-31) [7a04a4bf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:15:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-31) [7a04a4bf] FINISH, ActivateVdsVDSCommand, log id: 5fbe4781 >2013-03-11 04:15:00,221 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-31) [7a04a4bf] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:15:00,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-31) [7a04a4bf] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 04:15:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-31) [7a04a4bf] Autorecovering 1 storage domains >2013-03-11 04:15:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-31) [7a04a4bf] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 04:15:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-31) [751c64c6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:15:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-31) [751c64c6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:15 AM >2013-03-11 04:15:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-31) [751c64c6] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:15 AM >2013-03-11 04:15:01,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [13e84359] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6d2298a7 >2013-03-11 04:15:01,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [13e84359] FINISH, GetHardwareInfoVDSCommand, log id: 6d2298a7 >2013-03-11 04:15:01,432 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-18) [1a64fc35] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 04:15:01,435 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-18) [5910c60b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 04:15:01,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: d1da354 >2013-03-11 04:15:06,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 04:15:06,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:15:06,418 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 04:15:06,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: d1da354 >2013-03-11 04:15:06,422 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [7a63ab92] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:15:06,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [7a63ab92] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 716209ed >2013-03-11 04:15:06,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [7a63ab92] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 04:15:06,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [7a63ab92] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 716209ed >2013-03-11 04:15:06,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [7a63ab92] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 04:15:06,438 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [5910c60b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 04:15:06,447 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-18) [382ac4e8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:15:06,448 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [382ac4e8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 68671808 >2013-03-11 04:15:06,449 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [382ac4e8] FINISH, SetVdsStatusVDSCommand, log id: 68671808 >2013-03-11 04:15:06,455 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [382ac4e8] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 04:15:06,479 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [78a1e61f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:15:06,481 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [6110a170] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:15:06,482 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) [6110a170] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 04:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [71240b2] Autorecovering 1 hosts >2013-03-11 04:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [71240b2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-4) [1d2dab79] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-4) [1d2dab79] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:20:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [1d2dab79] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4bb38b19 >2013-03-11 04:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [1d2dab79] FINISH, SetVdsStatusVDSCommand, log id: 4bb38b19 >2013-03-11 04:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-4) [1d2dab79] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 27cc4bfc >2013-03-11 04:20:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [1d2dab79] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 11096b0f >2013-03-11 04:20:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [1d2dab79] FINISH, GetHardwareInfoVDSCommand, log id: 11096b0f >2013-03-11 04:20:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-4) [153576f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:20:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-4) [577cd899] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:20:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-4) [577cd899] FINISH, ActivateVdsVDSCommand, log id: 27cc4bfc >2013-03-11 04:20:00,219 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-4) [577cd899] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:20:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-4) [577cd899] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 04:20:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [577cd899] Autorecovering 1 storage domains >2013-03-11 04:20:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [577cd899] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 04:20:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-4) [919b576] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:20:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-4) [919b576] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:20 AM >2013-03-11 04:20:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-4) [919b576] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:20 AM >2013-03-11 04:20:00,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [495520dc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6e71a990 >2013-03-11 04:20:00,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [495520dc] FINISH, GetHardwareInfoVDSCommand, log id: 6e71a990 >2013-03-11 04:20:00,759 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-86) [5ac6cc8] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 04:20:00,763 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-86) [81246ae] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 04:20:00,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 30c326 >2013-03-11 04:20:06,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 04:20:06,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:20:06,757 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 04:20:06,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 30c326 >2013-03-11 04:20:06,761 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [22718cbf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:20:06,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [22718cbf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6b37b58c >2013-03-11 04:20:06,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [22718cbf] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 04:20:06,775 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [22718cbf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6b37b58c >2013-03-11 04:20:06,777 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [22718cbf] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 04:20:06,777 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [81246ae] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 04:20:06,786 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-86) [725b7c61] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:20:06,788 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [725b7c61] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 48d2b365 >2013-03-11 04:20:06,789 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [725b7c61] FINISH, SetVdsStatusVDSCommand, log id: 48d2b365 >2013-03-11 04:20:06,794 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-86) [725b7c61] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 04:20:06,818 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-86) [363f8dd9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:20:06,820 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-86) [319b33a5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:20:06,822 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-86) [319b33a5] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 04:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [2e993d7d] Autorecovering 1 hosts >2013-03-11 04:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [2e993d7d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-82) [2c81b433] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-82) [2c81b433] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [2c81b433] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 75177272 >2013-03-11 04:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [2c81b433] FINISH, SetVdsStatusVDSCommand, log id: 75177272 >2013-03-11 04:25:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-82) [2c81b433] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3c673ba4 >2013-03-11 04:25:00,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [2c81b433] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4bfb3bdc >2013-03-11 04:25:00,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [2c81b433] FINISH, GetHardwareInfoVDSCommand, log id: 4bfb3bdc >2013-03-11 04:25:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-82) [72b98e3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:25:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-82) [35837ce7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:25:00,205 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-82) [35837ce7] FINISH, ActivateVdsVDSCommand, log id: 3c673ba4 >2013-03-11 04:25:00,225 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-82) [35837ce7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:25:00,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-82) [35837ce7] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 04:25:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [35837ce7] Autorecovering 1 storage domains >2013-03-11 04:25:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-82) [35837ce7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 04:25:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-82) [7fab2ed3] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:25:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-82) [7fab2ed3] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:25 AM >2013-03-11 04:25:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-82) [7fab2ed3] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:25 AM >2013-03-11 04:25:01,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [424759f8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2d47f4e1 >2013-03-11 04:25:01,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [424759f8] FINISH, GetHardwareInfoVDSCommand, log id: 2d47f4e1 >2013-03-11 04:25:01,107 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-2) [30b70bf2] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 04:25:01,111 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-2) [3df72a1e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 04:25:01,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1ecbdf00 >2013-03-11 04:25:08,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 04:25:08,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:25:08,109 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 04:25:08,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 1ecbdf00 >2013-03-11 04:25:08,113 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [4d7059eb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:25:08,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4d7059eb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 384f806c >2013-03-11 04:25:08,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [4d7059eb] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 04:25:08,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4d7059eb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 384f806c >2013-03-11 04:25:08,129 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [4d7059eb] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 04:25:08,129 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [3df72a1e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 04:25:08,139 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-2) [6af3b6d1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:25:08,140 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [6af3b6d1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7e829cf4 >2013-03-11 04:25:08,141 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [6af3b6d1] FINISH, SetVdsStatusVDSCommand, log id: 7e829cf4 >2013-03-11 04:25:08,149 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [6af3b6d1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 04:25:08,173 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [206b6e49] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:25:08,175 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [f1e7b08] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:25:08,177 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-2) [f1e7b08] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 04:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [1d1d7172] Autorecovering 1 hosts >2013-03-11 04:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [1d1d7172] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [3b05737] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [3b05737] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [3b05737] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 55644309 >2013-03-11 04:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-16) [3b05737] FINISH, SetVdsStatusVDSCommand, log id: 55644309 >2013-03-11 04:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [3b05737] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6fa5e909 >2013-03-11 04:30:00,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [3b05737] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 53fe4344 >2013-03-11 04:30:00,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-16) [3b05737] FINISH, GetHardwareInfoVDSCommand, log id: 53fe4344 >2013-03-11 04:30:00,267 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-16) [79105966] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:30:00,270 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-16) [1f8c5c8d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:30:00,271 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-16) [1f8c5c8d] FINISH, ActivateVdsVDSCommand, log id: 6fa5e909 >2013-03-11 04:30:00,290 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-16) [1f8c5c8d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:30:00,291 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-16) [1f8c5c8d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 04:30:00,292 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [1f8c5c8d] Autorecovering 1 storage domains >2013-03-11 04:30:00,292 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-16) [1f8c5c8d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 04:30:00,293 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [38fc60d6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:30:00,293 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [38fc60d6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:30 AM >2013-03-11 04:30:00,296 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-16) [38fc60d6] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:30 AM >2013-03-11 04:30:02,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [5938953] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 56b3d59e >2013-03-11 04:30:02,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [5938953] FINISH, GetHardwareInfoVDSCommand, log id: 56b3d59e >2013-03-11 04:30:02,456 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-20) [39c0bbab] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 04:30:02,460 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-20) [56079994] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 04:30:02,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6db074ca >2013-03-11 04:30:08,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 04:30:08,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:30:08,089 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 04:30:08,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 6db074ca >2013-03-11 04:30:08,092 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [2db7d475] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:30:08,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [2db7d475] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7f94e42b >2013-03-11 04:30:08,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [2db7d475] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 04:30:08,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [2db7d475] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7f94e42b >2013-03-11 04:30:08,108 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [2db7d475] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 04:30:08,109 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [56079994] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 04:30:08,118 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-20) [1a53ffea] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:30:08,119 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [1a53ffea] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 26f27c9 >2013-03-11 04:30:08,121 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [1a53ffea] FINISH, SetVdsStatusVDSCommand, log id: 26f27c9 >2013-03-11 04:30:08,126 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-20) [1a53ffea] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 04:30:08,150 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-20) [3253598] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:30:08,153 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-20) [12380a29] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:30:08,154 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-20) [12380a29] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 04:30:33,871 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-11) [7d18a3ff] Start refreshing all users data >2013-03-11 04:30:35,051 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [f1e7b08] Attempting to update VMs/Templates Ovf. >2013-03-11 04:30:35,052 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [f1e7b08] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 04:30:35,053 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [f1e7b08] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 04:30:35,053 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [f1e7b08] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 04:30:35,054 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [f1e7b08] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 04:30:35,054 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [f1e7b08] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 04:30:35,055 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-2) [f1e7b08] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 04:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [80620b4] Autorecovering 1 hosts >2013-03-11 04:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [80620b4] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [746f35c1] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [746f35c1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [746f35c1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6e259e8b >2013-03-11 04:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [746f35c1] FINISH, SetVdsStatusVDSCommand, log id: 6e259e8b >2013-03-11 04:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [746f35c1] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 32261e5b >2013-03-11 04:35:00,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [746f35c1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4d8a2ba3 >2013-03-11 04:35:00,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [746f35c1] FINISH, GetHardwareInfoVDSCommand, log id: 4d8a2ba3 >2013-03-11 04:35:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [40d8449a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:35:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [526346fd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:35:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [526346fd] FINISH, ActivateVdsVDSCommand, log id: 32261e5b >2013-03-11 04:35:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [526346fd] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:35:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [526346fd] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 04:35:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [526346fd] Autorecovering 1 storage domains >2013-03-11 04:35:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [526346fd] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 04:35:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [20966fe5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:35:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [20966fe5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:35 AM >2013-03-11 04:35:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [20966fe5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:35 AM >2013-03-11 04:35:02,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [70173f2e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 289a9ab0 >2013-03-11 04:35:02,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [70173f2e] FINISH, GetHardwareInfoVDSCommand, log id: 289a9ab0 >2013-03-11 04:35:02,422 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-45) [679723a8] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 04:35:02,425 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-45) [39e71a3b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 04:35:02,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5e8a3811 >2013-03-11 04:35:09,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 04:35:09,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:35:09,432 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 04:35:09,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 5e8a3811 >2013-03-11 04:35:09,435 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [4091df52] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:35:09,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4091df52] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 11b8d5db >2013-03-11 04:35:09,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [4091df52] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 04:35:09,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4091df52] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 11b8d5db >2013-03-11 04:35:09,454 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [39e71a3b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 04:35:09,463 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-45) [7e4adf6d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:35:09,465 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [7e4adf6d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 191bb139 >2013-03-11 04:35:09,466 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [7e4adf6d] FINISH, SetVdsStatusVDSCommand, log id: 191bb139 >2013-03-11 04:35:09,472 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-45) [7e4adf6d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 04:35:09,496 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-45) [2cf72ed4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:35:09,498 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-45) [2733029b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:35:09,499 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-45) [2733029b] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 04:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [28cf91a] Autorecovering 1 hosts >2013-03-11 04:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [28cf91a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [77fe9611] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [77fe9611] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:40:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [77fe9611] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 41d718ff >2013-03-11 04:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [77fe9611] FINISH, SetVdsStatusVDSCommand, log id: 41d718ff >2013-03-11 04:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-89) [77fe9611] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5d19ddbb >2013-03-11 04:40:00,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [77fe9611] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 455e9555 >2013-03-11 04:40:00,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [77fe9611] FINISH, GetHardwareInfoVDSCommand, log id: 455e9555 >2013-03-11 04:40:00,255 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-89) [51319c70] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:40:00,258 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-89) [6e109777] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:40:00,259 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-89) [6e109777] FINISH, ActivateVdsVDSCommand, log id: 5d19ddbb >2013-03-11 04:40:00,278 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [6e109777] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:40:00,279 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-89) [6e109777] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 04:40:00,280 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [6e109777] Autorecovering 1 storage domains >2013-03-11 04:40:00,280 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [6e109777] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 04:40:00,281 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [402f6c35] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:40:00,281 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [402f6c35] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:40 AM >2013-03-11 04:40:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [402f6c35] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:40 AM >2013-03-11 04:40:00,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [7fab2ed3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1b2cab86 >2013-03-11 04:40:00,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [7fab2ed3] FINISH, GetHardwareInfoVDSCommand, log id: 1b2cab86 >2013-03-11 04:40:00,776 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-82) [46698b7b] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 04:40:00,779 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-82) [7ea61f17] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 04:40:00,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7efe3d9f >2013-03-11 04:40:06,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 04:40:06,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:40:06,776 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 04:40:06,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 7efe3d9f >2013-03-11 04:40:06,779 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [6cccecab] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:40:06,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6cccecab] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 75b737c6 >2013-03-11 04:40:06,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [6cccecab] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 04:40:06,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6cccecab] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 75b737c6 >2013-03-11 04:40:06,795 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [6cccecab] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 04:40:06,796 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [7ea61f17] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 04:40:06,805 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-82) [3cb4a361] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:40:06,807 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [3cb4a361] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 50cb203 >2013-03-11 04:40:06,808 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [3cb4a361] FINISH, SetVdsStatusVDSCommand, log id: 50cb203 >2013-03-11 04:40:06,817 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-82) [3cb4a361] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 04:40:06,840 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-82) [21091dd1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:40:06,842 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-82) [43cbe870] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:40:06,844 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-82) [43cbe870] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 04:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [f44a40d] Autorecovering 1 hosts >2013-03-11 04:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [f44a40d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [77de82e1] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [77de82e1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [77de82e1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4a74826e >2013-03-11 04:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [77de82e1] FINISH, SetVdsStatusVDSCommand, log id: 4a74826e >2013-03-11 04:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-88) [77de82e1] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 11dc2988 >2013-03-11 04:45:00,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [77de82e1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6daf825b >2013-03-11 04:45:00,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [77de82e1] FINISH, GetHardwareInfoVDSCommand, log id: 6daf825b >2013-03-11 04:45:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-88) [19485256] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:45:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-88) [b01b3b2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:45:00,207 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-88) [b01b3b2] FINISH, ActivateVdsVDSCommand, log id: 11dc2988 >2013-03-11 04:45:00,226 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [b01b3b2] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:45:00,227 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-88) [b01b3b2] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 04:45:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [b01b3b2] Autorecovering 1 storage domains >2013-03-11 04:45:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [b01b3b2] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 04:45:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [704d8ace] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:45:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [704d8ace] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:45 AM >2013-03-11 04:45:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [704d8ace] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:45 AM >2013-03-11 04:45:01,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [751c64c6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 76eaee96 >2013-03-11 04:45:01,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [751c64c6] FINISH, GetHardwareInfoVDSCommand, log id: 76eaee96 >2013-03-11 04:45:01,127 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-31) [595e3780] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 04:45:01,130 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-31) [4394862a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 04:45:01,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 12f2f871 >2013-03-11 04:45:07,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 04:45:07,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:45:07,167 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 04:45:07,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 12f2f871 >2013-03-11 04:45:07,170 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [54298b05] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:45:07,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [54298b05] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7252fe09 >2013-03-11 04:45:07,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [54298b05] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 04:45:07,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [54298b05] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7252fe09 >2013-03-11 04:45:07,186 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [54298b05] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 04:45:07,187 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [4394862a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 04:45:07,196 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-31) [e5c1ec2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:45:07,197 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [e5c1ec2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6e1d9347 >2013-03-11 04:45:07,199 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [e5c1ec2] FINISH, SetVdsStatusVDSCommand, log id: 6e1d9347 >2013-03-11 04:45:07,206 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-31) [e5c1ec2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 04:45:07,228 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-31) [606cc146] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:45:07,231 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-31) [65d12966] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:45:07,232 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-31) [65d12966] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 04:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [7c1b7f3c] Autorecovering 1 hosts >2013-03-11 04:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [7c1b7f3c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-91) [e16527c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-91) [e16527c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:50:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [e16527c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3acea520 >2013-03-11 04:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [e16527c] FINISH, SetVdsStatusVDSCommand, log id: 3acea520 >2013-03-11 04:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-91) [e16527c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6e901ea8 >2013-03-11 04:50:00,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [e16527c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 38e6c069 >2013-03-11 04:50:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [e16527c] FINISH, GetHardwareInfoVDSCommand, log id: 38e6c069 >2013-03-11 04:50:00,184 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-91) [5b1b0c3e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:50:00,187 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-91) [4c09f044] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:50:00,188 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-91) [4c09f044] FINISH, ActivateVdsVDSCommand, log id: 6e901ea8 >2013-03-11 04:50:00,208 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-91) [4c09f044] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:50:00,209 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-91) [4c09f044] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 04:50:00,210 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [4c09f044] Autorecovering 1 storage domains >2013-03-11 04:50:00,210 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [4c09f044] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 04:50:00,211 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-91) [66d774e0] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:50:00,211 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-91) [66d774e0] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:50 AM >2013-03-11 04:50:00,214 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-91) [66d774e0] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:50 AM >2013-03-11 04:50:01,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [39fe4430] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2c0b0b02 >2013-03-11 04:50:01,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [39fe4430] FINISH, GetHardwareInfoVDSCommand, log id: 2c0b0b02 >2013-03-11 04:50:01,512 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-68) [6da326f8] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 04:50:01,515 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-68) [43a5bfc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 04:50:01,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3d564f6e >2013-03-11 04:50:08,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 04:50:08,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:50:08,501 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 04:50:08,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 3d564f6e >2013-03-11 04:50:08,505 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [541f3ce4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:50:08,518 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [541f3ce4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 36fcb6f >2013-03-11 04:50:08,518 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [541f3ce4] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 04:50:08,518 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [541f3ce4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 36fcb6f >2013-03-11 04:50:08,520 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [541f3ce4] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 04:50:08,521 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [43a5bfc] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 04:50:08,530 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-68) [6f6b4bcc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:50:08,532 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [6f6b4bcc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2c80d58a >2013-03-11 04:50:08,533 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [6f6b4bcc] FINISH, SetVdsStatusVDSCommand, log id: 2c80d58a >2013-03-11 04:50:08,539 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-68) [6f6b4bcc] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 04:50:08,562 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-68) [3e84082b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:50:08,565 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-68) [20ebd7a8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:50:08,566 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-68) [20ebd7a8] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 04:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [56dad9e] Autorecovering 1 hosts >2013-03-11 04:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [56dad9e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [6c9a63b8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [6c9a63b8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [6c9a63b8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3ca6a026 >2013-03-11 04:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [6c9a63b8] FINISH, SetVdsStatusVDSCommand, log id: 3ca6a026 >2013-03-11 04:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-32) [6c9a63b8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6f2d096b >2013-03-11 04:55:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [6c9a63b8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 24e4dc8b >2013-03-11 04:55:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [6c9a63b8] FINISH, GetHardwareInfoVDSCommand, log id: 24e4dc8b >2013-03-11 04:55:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-32) [16b60ad3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:55:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-32) [58cf088b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:55:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-32) [58cf088b] FINISH, ActivateVdsVDSCommand, log id: 6f2d096b >2013-03-11 04:55:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [58cf088b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 04:55:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-32) [58cf088b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 04:55:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [58cf088b] Autorecovering 1 storage domains >2013-03-11 04:55:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [58cf088b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 04:55:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [7e4351f5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:55:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [7e4351f5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:55 AM >2013-03-11 04:55:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [7e4351f5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:55 AM >2013-03-11 04:55:02,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [402f6c35] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4db58551 >2013-03-11 04:55:02,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [402f6c35] FINISH, GetHardwareInfoVDSCommand, log id: 4db58551 >2013-03-11 04:55:02,847 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-89) [15e20778] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 04:55:02,851 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-89) [4779b3cb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 04:55:02,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2aef12d >2013-03-11 04:55:10,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 04:55:10,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 04:55:10,839 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 04:55:10,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 2aef12d >2013-03-11 04:55:10,843 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [1c0457b2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 04:55:10,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [1c0457b2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 146b5e9e >2013-03-11 04:55:10,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [1c0457b2] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 04:55:10,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [1c0457b2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 146b5e9e >2013-03-11 04:55:10,859 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [1c0457b2] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 04:55:10,860 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [4779b3cb] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 04:55:10,869 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-89) [6472201f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:55:10,871 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [6472201f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1bda6515 >2013-03-11 04:55:10,872 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [6472201f] FINISH, SetVdsStatusVDSCommand, log id: 1bda6515 >2013-03-11 04:55:10,878 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-89) [6472201f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 04:55:10,901 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-89) [3e4a6b19] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:55:10,903 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-89) [32a9e514] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 04:55:10,905 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-89) [32a9e514] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 05:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [148277f0] Autorecovering 1 hosts >2013-03-11 05:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [148277f0] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [57ce55a7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [57ce55a7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [57ce55a7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: f77ebd6 >2013-03-11 05:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [57ce55a7] FINISH, SetVdsStatusVDSCommand, log id: f77ebd6 >2013-03-11 05:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [57ce55a7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 727cf489 >2013-03-11 05:00:00,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [57ce55a7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 592fc392 >2013-03-11 05:00:00,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [57ce55a7] FINISH, GetHardwareInfoVDSCommand, log id: 592fc392 >2013-03-11 05:00:00,421 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [49bff908] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:00:00,424 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [5d3782e0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:00:00,425 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [5d3782e0] FINISH, ActivateVdsVDSCommand, log id: 727cf489 >2013-03-11 05:00:00,444 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [5d3782e0] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:00:00,445 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-8) [5d3782e0] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 05:00:00,447 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [5d3782e0] Autorecovering 1 storage domains >2013-03-11 05:00:00,447 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [5d3782e0] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 05:00:00,447 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [2cea74d1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:00:00,447 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [2cea74d1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:00 AM >2013-03-11 05:00:00,450 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [2cea74d1] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:00 AM >2013-03-11 05:00:02,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [1cf38a99] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 500d974 >2013-03-11 05:00:02,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [1cf38a99] FINISH, GetHardwareInfoVDSCommand, log id: 500d974 >2013-03-11 05:00:02,178 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-7) [73c05679] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 05:00:02,181 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-7) [621cb414] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 05:00:02,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5d5727a >2013-03-11 05:00:08,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 05:00:08,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:00:08,229 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 05:00:08,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 5d5727a >2013-03-11 05:00:08,232 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [2b74221f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:00:08,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [2b74221f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 92f0d73 >2013-03-11 05:00:08,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [2b74221f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 05:00:08,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [2b74221f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 92f0d73 >2013-03-11 05:00:08,248 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [2b74221f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 05:00:08,249 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [621cb414] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 05:00:08,258 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-7) [ae4ca42] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:00:08,260 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [ae4ca42] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3da3a082 >2013-03-11 05:00:08,261 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [ae4ca42] FINISH, SetVdsStatusVDSCommand, log id: 3da3a082 >2013-03-11 05:00:08,266 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-7) [ae4ca42] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 05:00:08,293 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-7) [53751a6f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:00:08,296 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-7) [32404736] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:00:08,297 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) [32404736] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 05:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [546f3764] Autorecovering 1 hosts >2013-03-11 05:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [546f3764] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-67) [45b6a73f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-67) [45b6a73f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:05:00,037 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [45b6a73f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1a3b1396 >2013-03-11 05:05:00,039 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [45b6a73f] FINISH, SetVdsStatusVDSCommand, log id: 1a3b1396 >2013-03-11 05:05:00,041 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-67) [45b6a73f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5c1149a >2013-03-11 05:05:00,197 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [45b6a73f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 55cbe81f >2013-03-11 05:05:00,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [45b6a73f] FINISH, GetHardwareInfoVDSCommand, log id: 55cbe81f >2013-03-11 05:05:00,235 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-67) [119d672c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:05:00,237 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-67) [5f62b2e7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:05:00,238 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-67) [5f62b2e7] FINISH, ActivateVdsVDSCommand, log id: 5c1149a >2013-03-11 05:05:00,258 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-67) [5f62b2e7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:05:00,261 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [5f62b2e7] Autorecovering 1 storage domains >2013-03-11 05:05:00,261 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-67) [5f62b2e7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 05:05:00,262 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-67) [27385058] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:05:00,262 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-67) [27385058] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:05 AM >2013-03-11 05:05:00,265 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-67) [27385058] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:05 AM >2013-03-11 05:05:02,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [21a7c7ab] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2b80fefa >2013-03-11 05:05:02,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [21a7c7ab] FINISH, GetHardwareInfoVDSCommand, log id: 2b80fefa >2013-03-11 05:05:02,568 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-49) [7412eb30] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 05:05:02,571 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-49) [3b315da1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 05:05:02,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 27fab451 >2013-03-11 05:05:09,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 05:05:09,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:05:09,574 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 05:05:09,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 27fab451 >2013-03-11 05:05:09,578 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [11abf741] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:05:09,592 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [11abf741] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 696b9b8a >2013-03-11 05:05:09,592 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [11abf741] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 05:05:09,592 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [11abf741] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 696b9b8a >2013-03-11 05:05:09,594 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [11abf741] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 05:05:09,595 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [3b315da1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 05:05:09,604 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-49) [3e9c208e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:05:09,605 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [3e9c208e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1c8fb96a >2013-03-11 05:05:09,607 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [3e9c208e] FINISH, SetVdsStatusVDSCommand, log id: 1c8fb96a >2013-03-11 05:05:09,613 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-49) [3e9c208e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 05:05:09,636 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-49) [5253bf6d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:05:09,638 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-49) [503f561c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:05:09,640 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) [503f561c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 05:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [442dd2eb] Autorecovering 1 hosts >2013-03-11 05:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [442dd2eb] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-28) [13fe6fc1] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-28) [13fe6fc1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [13fe6fc1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6c3099b6 >2013-03-11 05:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [13fe6fc1] FINISH, SetVdsStatusVDSCommand, log id: 6c3099b6 >2013-03-11 05:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-28) [13fe6fc1] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 648da07e >2013-03-11 05:10:00,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [13fe6fc1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 9b77fc >2013-03-11 05:10:00,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [13fe6fc1] FINISH, GetHardwareInfoVDSCommand, log id: 9b77fc >2013-03-11 05:10:00,404 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-28) [76a2e196] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:10:00,406 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-28) [b9940c8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:10:00,408 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-28) [b9940c8] FINISH, ActivateVdsVDSCommand, log id: 648da07e >2013-03-11 05:10:00,427 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-28) [b9940c8] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:10:00,428 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-28) [b9940c8] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 05:10:00,429 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [b9940c8] Autorecovering 1 storage domains >2013-03-11 05:10:00,429 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [b9940c8] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 05:10:00,430 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-28) [418fcc2f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:10:00,430 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-28) [418fcc2f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:10 AM >2013-03-11 05:10:00,433 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-28) [418fcc2f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:10 AM >2013-03-11 05:10:00,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [7cc06508] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 700d659d >2013-03-11 05:10:00,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [7cc06508] FINISH, GetHardwareInfoVDSCommand, log id: 700d659d >2013-03-11 05:10:00,917 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-80) [62557d03] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 05:10:00,920 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-80) [63861198] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 05:10:00,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: ca91dc8 >2013-03-11 05:10:06,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 05:10:06,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:10:06,919 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 05:10:06,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: ca91dc8 >2013-03-11 05:10:06,923 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [26bfdfae] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:10:06,936 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [26bfdfae] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 36ea28ee >2013-03-11 05:10:06,936 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [26bfdfae] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 05:10:06,936 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [26bfdfae] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 36ea28ee >2013-03-11 05:10:06,938 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [26bfdfae] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 05:10:06,939 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [63861198] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 05:10:06,948 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-80) [3769182b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:10:06,950 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [3769182b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 27a8ae93 >2013-03-11 05:10:06,951 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [3769182b] FINISH, SetVdsStatusVDSCommand, log id: 27a8ae93 >2013-03-11 05:10:06,957 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-80) [3769182b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 05:10:06,984 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-80) [79bdfc0d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:10:06,986 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-80) [12bd5363] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:10:06,988 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-80) [12bd5363] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 05:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [7e19e9ff] Autorecovering 1 hosts >2013-03-11 05:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [7e19e9ff] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [604052d1] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [604052d1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [604052d1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5583a14 >2013-03-11 05:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [604052d1] FINISH, SetVdsStatusVDSCommand, log id: 5583a14 >2013-03-11 05:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [604052d1] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 30cb278e >2013-03-11 05:15:01,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [604052d1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 676f696d >2013-03-11 05:15:01,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [604052d1] FINISH, GetHardwareInfoVDSCommand, log id: 676f696d >2013-03-11 05:15:01,057 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-63) [4db94df9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:15:01,060 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-63) [22b5ba37] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:15:01,061 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [22b5ba37] FINISH, ActivateVdsVDSCommand, log id: 30cb278e >2013-03-11 05:15:01,081 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [22b5ba37] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:15:01,082 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-63) [22b5ba37] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 05:15:01,083 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [22b5ba37] Autorecovering 1 storage domains >2013-03-11 05:15:01,083 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [22b5ba37] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 05:15:01,083 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [468728f1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:15:01,084 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [468728f1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:15 AM >2013-03-11 05:15:01,086 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [468728f1] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:15 AM >2013-03-11 05:15:01,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [1c38f254] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 19bb1c0b >2013-03-11 05:15:01,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [1c38f254] FINISH, GetHardwareInfoVDSCommand, log id: 19bb1c0b >2013-03-11 05:15:01,273 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-34) [348a0a5a] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 05:15:01,277 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-34) [3283468d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 05:15:01,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3e59ea88 >2013-03-11 05:15:07,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 05:15:07,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:15:07,293 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 05:15:07,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 3e59ea88 >2013-03-11 05:15:07,297 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [1bac0447] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:15:07,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [1bac0447] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: c8d3f91 >2013-03-11 05:15:07,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [1bac0447] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 05:15:07,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [1bac0447] FINISH, MarkPoolInReconstructModeVDSCommand, log id: c8d3f91 >2013-03-11 05:15:07,313 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [1bac0447] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 05:15:07,313 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [3283468d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 05:15:07,322 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-34) [3497d720] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:15:07,324 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [3497d720] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 112edcbb >2013-03-11 05:15:07,325 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [3497d720] FINISH, SetVdsStatusVDSCommand, log id: 112edcbb >2013-03-11 05:15:07,331 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-34) [3497d720] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 05:15:07,352 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-34) [3da9c2dd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:15:07,355 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-34) [15953ece] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:15:07,356 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) [15953ece] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 05:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [35c5f85c] Autorecovering 1 hosts >2013-03-11 05:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [35c5f85c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [2eb1ea21] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [2eb1ea21] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [2eb1ea21] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2970b777 >2013-03-11 05:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [2eb1ea21] FINISH, SetVdsStatusVDSCommand, log id: 2970b777 >2013-03-11 05:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [2eb1ea21] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5aebe374 >2013-03-11 05:20:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [2eb1ea21] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 746fd570 >2013-03-11 05:20:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [2eb1ea21] FINISH, GetHardwareInfoVDSCommand, log id: 746fd570 >2013-03-11 05:20:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-69) [691e1274] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:20:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-69) [77e04f63] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:20:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [77e04f63] FINISH, ActivateVdsVDSCommand, log id: 5aebe374 >2013-03-11 05:20:00,111 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [77e04f63] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:20:00,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-69) [77e04f63] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 05:20:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [77e04f63] Autorecovering 1 storage domains >2013-03-11 05:20:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [77e04f63] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 05:20:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [8d5b49c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:20:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [8d5b49c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:20 AM >2013-03-11 05:20:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [8d5b49c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:20 AM >2013-03-11 05:20:01,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [6eef8afb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 714f97a5 >2013-03-11 05:20:01,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [6eef8afb] FINISH, GetHardwareInfoVDSCommand, log id: 714f97a5 >2013-03-11 05:20:01,644 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-99) [564a8bf3] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 05:20:01,647 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-99) [6a2bd9ec] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 05:20:01,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 37bc206d >2013-03-11 05:20:08,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 05:20:08,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:20:08,640 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 05:20:08,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 37bc206d >2013-03-11 05:20:08,643 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [70d05f40] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:20:08,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [70d05f40] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4569894 >2013-03-11 05:20:08,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [70d05f40] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 05:20:08,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [70d05f40] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4569894 >2013-03-11 05:20:08,659 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [70d05f40] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 05:20:08,659 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [6a2bd9ec] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 05:20:08,669 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-99) [3b9c6579] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:20:08,670 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [3b9c6579] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 117c1b8f >2013-03-11 05:20:08,671 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [3b9c6579] FINISH, SetVdsStatusVDSCommand, log id: 117c1b8f >2013-03-11 05:20:08,677 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-99) [3b9c6579] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 05:20:08,705 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-99) [b1b6395] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:20:08,707 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-99) [3a30c61c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:20:08,709 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-99) [3a30c61c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 05:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [794e0400] Autorecovering 1 hosts >2013-03-11 05:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [794e0400] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-56) [21aed3c8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-56) [21aed3c8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [21aed3c8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1d81fc0f >2013-03-11 05:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [21aed3c8] FINISH, SetVdsStatusVDSCommand, log id: 1d81fc0f >2013-03-11 05:25:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-56) [21aed3c8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 70d87a04 >2013-03-11 05:25:00,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [21aed3c8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 17fe9788 >2013-03-11 05:25:00,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [21aed3c8] FINISH, GetHardwareInfoVDSCommand, log id: 17fe9788 >2013-03-11 05:25:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-56) [58b80106] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:25:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-56) [360d2035] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:25:00,205 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-56) [360d2035] FINISH, ActivateVdsVDSCommand, log id: 70d87a04 >2013-03-11 05:25:00,225 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-56) [360d2035] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:25:00,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-56) [360d2035] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 05:25:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [360d2035] Autorecovering 1 storage domains >2013-03-11 05:25:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [360d2035] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 05:25:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-56) [4acf5364] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:25:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-56) [4acf5364] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:25 AM >2013-03-11 05:25:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-56) [4acf5364] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:25 AM >2013-03-11 05:25:02,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [7bf8535c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 142683b6 >2013-03-11 05:25:02,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [7bf8535c] FINISH, GetHardwareInfoVDSCommand, log id: 142683b6 >2013-03-11 05:25:02,990 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-10) [745c8e60] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 05:25:02,993 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-10) [5927b18c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 05:25:03,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3fa088ff >2013-03-11 05:25:08,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 05:25:08,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:25:08,979 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 05:25:08,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 3fa088ff >2013-03-11 05:25:08,983 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [32b61ccf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:25:08,997 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [32b61ccf] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 49fcd122 >2013-03-11 05:25:08,997 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [32b61ccf] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 05:25:08,997 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [32b61ccf] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 49fcd122 >2013-03-11 05:25:08,999 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [32b61ccf] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 05:25:09,000 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [5927b18c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 05:25:09,009 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-10) [5b7742c0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:25:09,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [5b7742c0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 12098992 >2013-03-11 05:25:09,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [5b7742c0] FINISH, SetVdsStatusVDSCommand, log id: 12098992 >2013-03-11 05:25:09,018 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-10) [5b7742c0] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 05:25:09,040 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-10) [1197577d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:25:09,043 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-10) [342f8a71] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:25:09,044 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-10) [342f8a71] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 05:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [655d7eb] Autorecovering 1 hosts >2013-03-11 05:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [655d7eb] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-38) [492190f3] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-38) [492190f3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [492190f3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7e802ef1 >2013-03-11 05:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [492190f3] FINISH, SetVdsStatusVDSCommand, log id: 7e802ef1 >2013-03-11 05:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-38) [492190f3] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: ca1b456 >2013-03-11 05:30:00,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [492190f3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 30a43d9f >2013-03-11 05:30:00,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [492190f3] FINISH, GetHardwareInfoVDSCommand, log id: 30a43d9f >2013-03-11 05:30:00,273 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-38) [340dcc3c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:30:00,275 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-38) [4253361f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:30:00,276 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-38) [4253361f] FINISH, ActivateVdsVDSCommand, log id: ca1b456 >2013-03-11 05:30:00,296 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-38) [4253361f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:30:00,297 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-38) [4253361f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 05:30:00,298 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [4253361f] Autorecovering 1 storage domains >2013-03-11 05:30:00,298 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [4253361f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 05:30:00,299 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-38) [1d4ef37d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:30:00,299 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-38) [1d4ef37d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:30 AM >2013-03-11 05:30:00,302 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-38) [1d4ef37d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:30 AM >2013-03-11 05:30:00,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [25427d38] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7be5d06d >2013-03-11 05:30:00,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [25427d38] FINISH, GetHardwareInfoVDSCommand, log id: 7be5d06d >2013-03-11 05:30:00,373 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-97) [12a1dd40] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 05:30:00,377 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-97) [154078b7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 05:30:00,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7030c3d9 >2013-03-11 05:30:06,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 05:30:06,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:30:06,467 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 05:30:06,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 7030c3d9 >2013-03-11 05:30:06,471 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [686be546] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:30:06,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [686be546] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 785e30d1 >2013-03-11 05:30:06,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [686be546] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 05:30:06,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [686be546] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 785e30d1 >2013-03-11 05:30:06,486 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [686be546] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 05:30:06,487 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [154078b7] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 05:30:06,498 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-97) [28bbb95a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:30:06,499 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [28bbb95a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 478a10d1 >2013-03-11 05:30:06,500 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [28bbb95a] FINISH, SetVdsStatusVDSCommand, log id: 478a10d1 >2013-03-11 05:30:06,506 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-97) [28bbb95a] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 05:30:06,529 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-97) [4e8a2ad4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:30:06,532 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-97) [68e3aa6a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:30:06,533 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-97) [68e3aa6a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 05:30:33,872 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-50) [9afc96e] Start refreshing all users data >2013-03-11 05:30:35,056 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [12bd5363] Attempting to update VMs/Templates Ovf. >2013-03-11 05:30:35,056 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [12bd5363] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 05:30:35,057 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [12bd5363] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 05:30:35,057 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [12bd5363] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 05:30:35,058 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [12bd5363] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 05:30:35,058 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [12bd5363] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 05:30:35,059 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-80) [12bd5363] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 05:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [8d5b49c] Autorecovering 1 hosts >2013-03-11 05:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [8d5b49c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [6a14d06c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [6a14d06c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [6a14d06c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1cf43452 >2013-03-11 05:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [6a14d06c] FINISH, SetVdsStatusVDSCommand, log id: 1cf43452 >2013-03-11 05:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [6a14d06c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 69f85fbd >2013-03-11 05:35:00,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [6a14d06c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 8a22175 >2013-03-11 05:35:00,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [6a14d06c] FINISH, GetHardwareInfoVDSCommand, log id: 8a22175 >2013-03-11 05:35:00,261 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-69) [763e00c7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:35:00,263 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-69) [1059fef7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:35:00,264 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [1059fef7] FINISH, ActivateVdsVDSCommand, log id: 69f85fbd >2013-03-11 05:35:00,284 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [1059fef7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:35:00,285 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-69) [1059fef7] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 05:35:00,286 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [1059fef7] Autorecovering 1 storage domains >2013-03-11 05:35:00,286 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [1059fef7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 05:35:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [5dd9492e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:35:00,287 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [5dd9492e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:35 AM >2013-03-11 05:35:00,289 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [5dd9492e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:35 AM >2013-03-11 05:35:00,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [612d1d72] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4e397a25 >2013-03-11 05:35:00,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [612d1d72] FINISH, GetHardwareInfoVDSCommand, log id: 4e397a25 >2013-03-11 05:35:00,813 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-46) [349a022e] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 05:35:00,817 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-46) [64c08915] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 05:35:00,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1273f085 >2013-03-11 05:35:07,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 05:35:07,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:35:07,812 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 05:35:07,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 1273f085 >2013-03-11 05:35:07,816 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [57b3ff4e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:35:07,834 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [57b3ff4e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2b8b66fe >2013-03-11 05:35:07,834 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [57b3ff4e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 05:35:07,834 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [57b3ff4e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2b8b66fe >2013-03-11 05:35:07,836 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [57b3ff4e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 05:35:07,836 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [64c08915] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 05:35:07,846 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-46) [1140ea6c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:35:07,848 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [1140ea6c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 55b3bd02 >2013-03-11 05:35:07,849 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [1140ea6c] FINISH, SetVdsStatusVDSCommand, log id: 55b3bd02 >2013-03-11 05:35:07,854 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-46) [1140ea6c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 05:35:07,878 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-46) [938faab] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:35:07,880 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-46) [4d79dedf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:35:07,882 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-46) [4d79dedf] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 05:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [1c0887d6] Autorecovering 1 hosts >2013-03-11 05:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [1c0887d6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [78584e84] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [78584e84] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [78584e84] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7393ee62 >2013-03-11 05:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [78584e84] FINISH, SetVdsStatusVDSCommand, log id: 7393ee62 >2013-03-11 05:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-41) [78584e84] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1df0e6cb >2013-03-11 05:40:00,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [78584e84] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2d4b780e >2013-03-11 05:40:00,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [78584e84] FINISH, GetHardwareInfoVDSCommand, log id: 2d4b780e >2013-03-11 05:40:00,187 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-41) [721498ea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:40:00,189 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-41) [7e605d31] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:40:00,191 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-41) [7e605d31] FINISH, ActivateVdsVDSCommand, log id: 1df0e6cb >2013-03-11 05:40:00,210 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-41) [7e605d31] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:40:00,211 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-41) [7e605d31] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 05:40:00,212 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [7e605d31] Autorecovering 1 storage domains >2013-03-11 05:40:00,212 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-41) [7e605d31] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 05:40:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [2b0dfecd] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:40:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [2b0dfecd] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:40 AM >2013-03-11 05:40:00,216 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-41) [2b0dfecd] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:40 AM >2013-03-11 05:40:02,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [418fcc2f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6889ea41 >2013-03-11 05:40:02,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [418fcc2f] FINISH, GetHardwareInfoVDSCommand, log id: 6889ea41 >2013-03-11 05:40:02,166 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-28) [23215a76] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 05:40:02,170 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-28) [1ab68fb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 05:40:02,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6eb40e20 >2013-03-11 05:40:07,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 05:40:07,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:40:07,176 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 05:40:07,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 6eb40e20 >2013-03-11 05:40:07,180 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [5e681525] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:40:07,194 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5e681525] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2faf72c6 >2013-03-11 05:40:07,194 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [5e681525] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 05:40:07,194 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5e681525] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2faf72c6 >2013-03-11 05:40:07,199 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [1ab68fb] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 05:40:07,208 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-28) [a501819] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:40:07,210 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [a501819] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 14857bde >2013-03-11 05:40:07,211 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [a501819] FINISH, SetVdsStatusVDSCommand, log id: 14857bde >2013-03-11 05:40:07,217 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-28) [a501819] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 05:40:07,245 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-28) [3f7ccd71] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:40:07,248 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-28) [23de5e2c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:40:07,249 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-28) [23de5e2c] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 05:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [468728f1] Autorecovering 1 hosts >2013-03-11 05:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [468728f1] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [5ab2fd25] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [5ab2fd25] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [5ab2fd25] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 376dc348 >2013-03-11 05:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [5ab2fd25] FINISH, SetVdsStatusVDSCommand, log id: 376dc348 >2013-03-11 05:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [5ab2fd25] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4a69f2ea >2013-03-11 05:45:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [5ab2fd25] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1a016bb7 >2013-03-11 05:45:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [5ab2fd25] FINISH, GetHardwareInfoVDSCommand, log id: 1a016bb7 >2013-03-11 05:45:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-63) [6a1f5c93] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:45:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-63) [7e83d048] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:45:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [7e83d048] FINISH, ActivateVdsVDSCommand, log id: 4a69f2ea >2013-03-11 05:45:00,219 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [7e83d048] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:45:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-63) [7e83d048] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 05:45:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [7e83d048] Autorecovering 1 storage domains >2013-03-11 05:45:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [7e83d048] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 05:45:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [43c55d8a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:45:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [43c55d8a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:45 AM >2013-03-11 05:45:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [43c55d8a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:45 AM >2013-03-11 05:45:01,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [59d2b06c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7ae35644 >2013-03-11 05:45:01,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [59d2b06c] FINISH, GetHardwareInfoVDSCommand, log id: 7ae35644 >2013-03-11 05:45:01,535 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-47) [4a59a07e] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 05:45:01,538 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-47) [2d7560d6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 05:45:01,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 34371783 >2013-03-11 05:45:08,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 05:45:08,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:45:08,933 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 05:45:08,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 34371783 >2013-03-11 05:45:08,937 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [26c5150] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:45:08,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [26c5150] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2b424859 >2013-03-11 05:45:08,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [26c5150] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 05:45:08,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [26c5150] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2b424859 >2013-03-11 05:45:08,953 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [26c5150] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 05:45:08,953 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [2d7560d6] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 05:45:08,963 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-47) [e1c3c64] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:45:08,965 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [e1c3c64] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 303c34d0 >2013-03-11 05:45:08,966 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [e1c3c64] FINISH, SetVdsStatusVDSCommand, log id: 303c34d0 >2013-03-11 05:45:08,972 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [e1c3c64] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 05:45:08,995 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [3da5db05] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:45:08,998 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [db67848] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:45:08,999 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-47) [db67848] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 05:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [32116524] Autorecovering 1 hosts >2013-03-11 05:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [32116524] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [211a867] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [211a867] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [211a867] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4dbb0a25 >2013-03-11 05:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [211a867] FINISH, SetVdsStatusVDSCommand, log id: 4dbb0a25 >2013-03-11 05:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-19) [211a867] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 35db49b2 >2013-03-11 05:50:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [211a867] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 553d867d >2013-03-11 05:50:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [211a867] FINISH, GetHardwareInfoVDSCommand, log id: 553d867d >2013-03-11 05:50:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-19) [ef387a6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:50:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-19) [1188ba87] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:50:00,203 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-19) [1188ba87] FINISH, ActivateVdsVDSCommand, log id: 35db49b2 >2013-03-11 05:50:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [1188ba87] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:50:00,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-19) [1188ba87] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 05:50:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [1188ba87] Autorecovering 1 storage domains >2013-03-11 05:50:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [1188ba87] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 05:50:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [2c9b0dc9] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:50:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [2c9b0dc9] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:50 AM >2013-03-11 05:50:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [2c9b0dc9] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:50 AM >2013-03-11 05:50:00,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [2a001dd4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2955d3db >2013-03-11 05:50:00,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [2a001dd4] FINISH, GetHardwareInfoVDSCommand, log id: 2955d3db >2013-03-11 05:50:00,300 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-12) [7779c7fb] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 05:50:00,304 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-12) [77fbf38d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 05:50:00,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 43cf6197 >2013-03-11 05:50:06,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 05:50:06,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:50:06,296 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 05:50:06,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 43cf6197 >2013-03-11 05:50:06,300 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [45033332] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:50:06,314 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [45033332] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6e658ed7 >2013-03-11 05:50:06,314 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [45033332] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 05:50:06,314 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [45033332] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6e658ed7 >2013-03-11 05:50:06,316 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [45033332] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 05:50:06,316 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [77fbf38d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 05:50:06,326 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-12) [517ff5b9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:50:06,327 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [517ff5b9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 110e53f5 >2013-03-11 05:50:06,328 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [517ff5b9] FINISH, SetVdsStatusVDSCommand, log id: 110e53f5 >2013-03-11 05:50:06,334 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-12) [517ff5b9] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 05:50:06,358 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-12) [22089981] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:50:06,360 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-12) [ab15a99] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:50:06,361 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-12) [ab15a99] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 05:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [260a8c0d] Autorecovering 1 hosts >2013-03-11 05:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [260a8c0d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-98) [76ebee02] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-98) [76ebee02] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-98) [76ebee02] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 122d5c47 >2013-03-11 05:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-98) [76ebee02] FINISH, SetVdsStatusVDSCommand, log id: 122d5c47 >2013-03-11 05:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-98) [76ebee02] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4ce3bb7d >2013-03-11 05:55:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-98) [76ebee02] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 11ec9098 >2013-03-11 05:55:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-98) [76ebee02] FINISH, GetHardwareInfoVDSCommand, log id: 11ec9098 >2013-03-11 05:55:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-98) [54554e40] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:55:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-98) [4d644fc0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:55:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-98) [4d644fc0] FINISH, ActivateVdsVDSCommand, log id: 4ce3bb7d >2013-03-11 05:55:00,222 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-98) [4d644fc0] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 05:55:00,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-98) [4d644fc0] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 05:55:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [4d644fc0] Autorecovering 1 storage domains >2013-03-11 05:55:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-98) [4d644fc0] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 05:55:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-98) [3365b147] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:55:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-98) [3365b147] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:55 AM >2013-03-11 05:55:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-98) [3365b147] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:55 AM >2013-03-11 05:55:00,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [3d5118cd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 58b49e78 >2013-03-11 05:55:00,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [3d5118cd] FINISH, GetHardwareInfoVDSCommand, log id: 58b49e78 >2013-03-11 05:55:00,642 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-65) [49820bd3] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 05:55:00,646 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-65) [57005af2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 05:55:00,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2d0f8c10 >2013-03-11 05:55:05,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 05:55:05,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 05:55:05,624 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 05:55:05,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 2d0f8c10 >2013-03-11 05:55:05,628 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [4d1bcbb4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 05:55:05,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4d1bcbb4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7740bea5 >2013-03-11 05:55:05,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [4d1bcbb4] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 05:55:05,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4d1bcbb4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7740bea5 >2013-03-11 05:55:05,644 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [4d1bcbb4] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 05:55:05,644 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [57005af2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 05:55:05,654 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-65) [143667bf] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:55:05,656 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [143667bf] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3a5ec11b >2013-03-11 05:55:05,657 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [143667bf] FINISH, SetVdsStatusVDSCommand, log id: 3a5ec11b >2013-03-11 05:55:05,663 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-65) [143667bf] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 05:55:05,686 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-65) [2e7864bf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:55:05,689 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-65) [3997f98] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 05:55:05,690 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-65) [3997f98] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 06:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [124dcd67] Autorecovering 1 hosts >2013-03-11 06:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [124dcd67] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-74) [687e2523] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-74) [687e2523] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [687e2523] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 403414c5 >2013-03-11 06:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [687e2523] FINISH, SetVdsStatusVDSCommand, log id: 403414c5 >2013-03-11 06:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-74) [687e2523] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 51ab47a5 >2013-03-11 06:00:00,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [687e2523] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 322cd34e >2013-03-11 06:00:00,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [687e2523] FINISH, GetHardwareInfoVDSCommand, log id: 322cd34e >2013-03-11 06:00:00,265 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-74) [349f8a37] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:00:00,268 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-74) [ddc06d7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:00:00,269 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-74) [ddc06d7] FINISH, ActivateVdsVDSCommand, log id: 51ab47a5 >2013-03-11 06:00:00,288 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-74) [ddc06d7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:00:00,289 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-74) [ddc06d7] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 06:00:00,290 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [ddc06d7] Autorecovering 1 storage domains >2013-03-11 06:00:00,290 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [ddc06d7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 06:00:00,291 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-74) [5c4b2130] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:00:00,291 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-74) [5c4b2130] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:00 AM >2013-03-11 06:00:00,294 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-74) [5c4b2130] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:00 AM >2013-03-11 06:00:02,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [f501cee] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 71d8b737 >2013-03-11 06:00:02,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [f501cee] FINISH, GetHardwareInfoVDSCommand, log id: 71d8b737 >2013-03-11 06:00:02,975 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-64) [2d41de75] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 06:00:02,979 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-64) [4425cd8e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 06:00:02,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 73779851 >2013-03-11 06:00:10,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 06:00:10,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:00:10,024 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 06:00:10,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 73779851 >2013-03-11 06:00:10,028 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [57fb4968] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:00:10,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [57fb4968] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 19fe6bd4 >2013-03-11 06:00:10,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [57fb4968] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 06:00:10,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [57fb4968] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 19fe6bd4 >2013-03-11 06:00:10,043 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [57fb4968] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 06:00:10,044 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [4425cd8e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 06:00:10,053 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-64) [41434cef] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:00:10,055 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [41434cef] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 489db037 >2013-03-11 06:00:10,056 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [41434cef] FINISH, SetVdsStatusVDSCommand, log id: 489db037 >2013-03-11 06:00:10,062 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-64) [41434cef] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 06:00:10,084 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-64) [7ad1141] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:00:10,086 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-64) [27d2a58d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:00:10,087 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-64) [27d2a58d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 06:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [14967fe1] Autorecovering 1 hosts >2013-03-11 06:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [14967fe1] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-60) [1c33c7f8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-60) [1c33c7f8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [1c33c7f8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 22877fd1 >2013-03-11 06:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [1c33c7f8] FINISH, SetVdsStatusVDSCommand, log id: 22877fd1 >2013-03-11 06:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-60) [1c33c7f8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4fb1a2ca >2013-03-11 06:05:00,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [1c33c7f8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4ef06d7 >2013-03-11 06:05:00,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [1c33c7f8] FINISH, GetHardwareInfoVDSCommand, log id: 4ef06d7 >2013-03-11 06:05:00,263 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-60) [1bfd4ea2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:05:00,265 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-60) [1bb313dd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:05:00,266 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-60) [1bb313dd] FINISH, ActivateVdsVDSCommand, log id: 4fb1a2ca >2013-03-11 06:05:00,286 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-60) [1bb313dd] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:05:00,287 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-60) [1bb313dd] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 06:05:00,288 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [1bb313dd] Autorecovering 1 storage domains >2013-03-11 06:05:00,288 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [1bb313dd] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 06:05:00,289 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-60) [5142ccc] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:05:00,289 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-60) [5142ccc] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:05 AM >2013-03-11 06:05:00,291 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-60) [5142ccc] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:05 AM >2013-03-11 06:05:01,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [2c9b0dc9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5c91c058 >2013-03-11 06:05:01,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [2c9b0dc9] FINISH, GetHardwareInfoVDSCommand, log id: 5c91c058 >2013-03-11 06:05:01,372 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-19) [7eaaece8] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 06:05:01,376 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-19) [5a54f7f3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 06:05:01,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 726ad8ac >2013-03-11 06:05:06,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 06:05:06,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:05:06,377 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 06:05:06,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 726ad8ac >2013-03-11 06:05:06,381 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [4ac6c54f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:05:06,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4ac6c54f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 646a4390 >2013-03-11 06:05:06,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [4ac6c54f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 06:05:06,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [4ac6c54f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 646a4390 >2013-03-11 06:05:06,396 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [4ac6c54f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 06:05:06,397 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [5a54f7f3] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 06:05:06,406 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-19) [2d7d49db] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:05:06,408 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [2d7d49db] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 52a4ff48 >2013-03-11 06:05:06,409 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [2d7d49db] FINISH, SetVdsStatusVDSCommand, log id: 52a4ff48 >2013-03-11 06:05:06,415 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-19) [2d7d49db] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 06:05:06,438 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-19) [5926aeac] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:05:06,441 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-19) [1c369396] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:05:06,442 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-19) [1c369396] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 06:10:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [337c7da4] Autorecovering 1 hosts >2013-03-11 06:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [337c7da4] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [5293d33c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [5293d33c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [5293d33c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5234b4ca >2013-03-11 06:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [5293d33c] FINISH, SetVdsStatusVDSCommand, log id: 5234b4ca >2013-03-11 06:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-77) [5293d33c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6ec2ec91 >2013-03-11 06:10:00,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [5293d33c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 557299ec >2013-03-11 06:10:00,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [5293d33c] FINISH, GetHardwareInfoVDSCommand, log id: 557299ec >2013-03-11 06:10:00,412 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-77) [42c90a0f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:10:00,414 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-77) [554e513e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:10:00,415 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-77) [554e513e] FINISH, ActivateVdsVDSCommand, log id: 6ec2ec91 >2013-03-11 06:10:00,435 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [554e513e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:10:00,436 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-77) [554e513e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 06:10:00,437 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [554e513e] Autorecovering 1 storage domains >2013-03-11 06:10:00,437 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [554e513e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 06:10:00,438 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [5b2ea147] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:10:00,438 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [5b2ea147] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:10 AM >2013-03-11 06:10:00,477 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [5b2ea147] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:10 AM >2013-03-11 06:10:00,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [1edc5e4b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6af5883 >2013-03-11 06:10:00,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [1edc5e4b] FINISH, GetHardwareInfoVDSCommand, log id: 6af5883 >2013-03-11 06:10:00,720 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-53) [7729b617] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 06:10:00,723 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-53) [7e30451c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 06:10:00,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2f51a13 >2013-03-11 06:10:06,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 06:10:06,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:10:06,724 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 06:10:06,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2f51a13 >2013-03-11 06:10:06,728 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [c267db5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:10:06,741 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [c267db5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4de6f822 >2013-03-11 06:10:06,741 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [c267db5] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 06:10:06,741 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [c267db5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4de6f822 >2013-03-11 06:10:06,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [c267db5] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 06:10:06,744 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [7e30451c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 06:10:06,753 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-53) [7cdbeadd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:10:06,755 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [7cdbeadd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6c45f835 >2013-03-11 06:10:06,756 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [7cdbeadd] FINISH, SetVdsStatusVDSCommand, log id: 6c45f835 >2013-03-11 06:10:06,762 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-53) [7cdbeadd] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 06:10:06,784 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-53) [2b95a91] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:10:06,787 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-53) [e6ba8a7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:10:06,789 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-53) [e6ba8a7] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 06:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) [41f2f67a] Autorecovering 1 hosts >2013-03-11 06:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) [41f2f67a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-62) [4dd53a38] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-62) [4dd53a38] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-62) [4dd53a38] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 30363494 >2013-03-11 06:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-62) [4dd53a38] FINISH, SetVdsStatusVDSCommand, log id: 30363494 >2013-03-11 06:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-62) [4dd53a38] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 74620c33 >2013-03-11 06:15:00,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-62) [4dd53a38] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 65b51d8c >2013-03-11 06:15:00,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-62) [4dd53a38] FINISH, GetHardwareInfoVDSCommand, log id: 65b51d8c >2013-03-11 06:15:00,256 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-62) [555ee44c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:15:00,258 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-62) [72722f6b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:15:00,260 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-62) [72722f6b] FINISH, ActivateVdsVDSCommand, log id: 74620c33 >2013-03-11 06:15:00,279 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-62) [72722f6b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:15:00,280 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-62) [72722f6b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 06:15:00,281 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) [72722f6b] Autorecovering 1 storage domains >2013-03-11 06:15:00,281 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-62) [72722f6b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 06:15:00,282 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-62) [6354c15d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:15:00,282 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-62) [6354c15d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:15 AM >2013-03-11 06:15:00,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-62) [6354c15d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:15 AM >2013-03-11 06:15:01,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [5c4b2130] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5408db07 >2013-03-11 06:15:01,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [5c4b2130] FINISH, GetHardwareInfoVDSCommand, log id: 5408db07 >2013-03-11 06:15:01,074 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-74) [5c2f4f01] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 06:15:01,077 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-74) [58ce108d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 06:15:01,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1eb0d0eb >2013-03-11 06:15:07,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 06:15:07,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:15:07,071 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 06:15:07,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 1eb0d0eb >2013-03-11 06:15:07,075 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [5d0cadc9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:15:07,092 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5d0cadc9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1a941e1d >2013-03-11 06:15:07,092 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [5d0cadc9] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 06:15:07,092 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5d0cadc9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1a941e1d >2013-03-11 06:15:07,094 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [5d0cadc9] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 06:15:07,094 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [58ce108d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 06:15:07,104 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-74) [73d9b980] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:15:07,105 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [73d9b980] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4b06cb34 >2013-03-11 06:15:07,106 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [73d9b980] FINISH, SetVdsStatusVDSCommand, log id: 4b06cb34 >2013-03-11 06:15:07,112 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-74) [73d9b980] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 06:15:07,140 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-74) [10666a69] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:15:07,142 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-74) [4708f8ed] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:15:07,144 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-74) [4708f8ed] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 06:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) [7675f75] Autorecovering 1 hosts >2013-03-11 06:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) [7675f75] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-30) [4b32291] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-30) [4b32291] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-30) [4b32291] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 762c89cf >2013-03-11 06:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-30) [4b32291] FINISH, SetVdsStatusVDSCommand, log id: 762c89cf >2013-03-11 06:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-30) [4b32291] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1e5177e9 >2013-03-11 06:20:00,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-30) [4b32291] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 218776d6 >2013-03-11 06:20:00,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-30) [4b32291] FINISH, GetHardwareInfoVDSCommand, log id: 218776d6 >2013-03-11 06:20:00,407 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-30) [37afa3da] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:20:00,410 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-30) [efb7e93] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:20:00,411 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-30) [efb7e93] FINISH, ActivateVdsVDSCommand, log id: 1e5177e9 >2013-03-11 06:20:00,430 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-30) [efb7e93] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:20:00,431 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-30) [efb7e93] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 06:20:00,432 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) [efb7e93] Autorecovering 1 storage domains >2013-03-11 06:20:00,432 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) [efb7e93] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 06:20:00,433 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-30) [5cb2d43f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:20:00,433 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-30) [5cb2d43f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:20 AM >2013-03-11 06:20:00,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-30) [5cb2d43f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:20 AM >2013-03-11 06:20:01,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [1ea9775] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2fdf8cde >2013-03-11 06:20:01,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [1ea9775] FINISH, GetHardwareInfoVDSCommand, log id: 2fdf8cde >2013-03-11 06:20:01,427 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-54) [26ab49e9] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 06:20:01,431 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-54) [3cf4036] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 06:20:01,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 66a0a20c >2013-03-11 06:20:08,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 06:20:08,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:20:08,407 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 06:20:08,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 66a0a20c >2013-03-11 06:20:08,411 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [5184737e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:20:08,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [5184737e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4fd55806 >2013-03-11 06:20:08,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [5184737e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 06:20:08,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [5184737e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4fd55806 >2013-03-11 06:20:08,426 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [5184737e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 06:20:08,427 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [3cf4036] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 06:20:08,436 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-54) [332c5de6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:20:08,437 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [332c5de6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: c354512 >2013-03-11 06:20:08,439 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [332c5de6] FINISH, SetVdsStatusVDSCommand, log id: c354512 >2013-03-11 06:20:08,446 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [332c5de6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 06:20:08,468 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [190e523b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:20:08,471 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [5e282629] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:20:08,472 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-54) [5e282629] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 06:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [64e9fe5f] Autorecovering 1 hosts >2013-03-11 06:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [64e9fe5f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [132caf9] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [132caf9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [132caf9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 70ff1ebe >2013-03-11 06:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [132caf9] FINISH, SetVdsStatusVDSCommand, log id: 70ff1ebe >2013-03-11 06:25:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [132caf9] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3a5d0c2d >2013-03-11 06:25:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [132caf9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7d16788c >2013-03-11 06:25:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [132caf9] FINISH, GetHardwareInfoVDSCommand, log id: 7d16788c >2013-03-11 06:25:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [3d77e5b0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:25:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [6d18d29b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:25:00,203 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [6d18d29b] FINISH, ActivateVdsVDSCommand, log id: 3a5d0c2d >2013-03-11 06:25:00,222 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [6d18d29b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:25:00,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [6d18d29b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 06:25:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [6d18d29b] Autorecovering 1 storage domains >2013-03-11 06:25:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [6d18d29b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 06:25:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [5904654d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:25:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [5904654d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:25 AM >2013-03-11 06:25:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [5904654d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:25 AM >2013-03-11 06:25:02,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [5b2ea147] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7b522d0e >2013-03-11 06:25:02,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [5b2ea147] FINISH, GetHardwareInfoVDSCommand, log id: 7b522d0e >2013-03-11 06:25:02,753 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-77) [656adaf9] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 06:25:02,756 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-77) [563e1279] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 06:25:02,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 762a6c20 >2013-03-11 06:25:07,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 06:25:07,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:25:07,345 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 06:25:07,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 762a6c20 >2013-03-11 06:25:07,349 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [551760ea] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:25:07,363 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [551760ea] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 57efaaa1 >2013-03-11 06:25:07,363 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [551760ea] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 06:25:07,363 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [551760ea] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 57efaaa1 >2013-03-11 06:25:07,365 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [551760ea] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 06:25:07,366 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [563e1279] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 06:25:07,375 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-77) [55a258c8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:25:07,376 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [55a258c8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 450d21cd >2013-03-11 06:25:07,378 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [55a258c8] FINISH, SetVdsStatusVDSCommand, log id: 450d21cd >2013-03-11 06:25:07,384 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-77) [55a258c8] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 06:25:07,406 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-77) [10135925] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:25:07,409 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-77) [37a992da] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:25:07,410 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-77) [37a992da] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 06:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [49fe93dc] Autorecovering 1 hosts >2013-03-11 06:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [49fe93dc] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-55) [57a4d7bc] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-55) [57a4d7bc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [57a4d7bc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5a3f1908 >2013-03-11 06:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [57a4d7bc] FINISH, SetVdsStatusVDSCommand, log id: 5a3f1908 >2013-03-11 06:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-55) [57a4d7bc] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 69166059 >2013-03-11 06:30:00,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [57a4d7bc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 147f2a50 >2013-03-11 06:30:00,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [57a4d7bc] FINISH, GetHardwareInfoVDSCommand, log id: 147f2a50 >2013-03-11 06:30:00,408 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-55) [75ee588e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:30:00,411 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-55) [3c4ee664] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:30:00,412 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-55) [3c4ee664] FINISH, ActivateVdsVDSCommand, log id: 69166059 >2013-03-11 06:30:00,431 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-55) [3c4ee664] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:30:00,432 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-55) [3c4ee664] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 06:30:00,433 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [3c4ee664] Autorecovering 1 storage domains >2013-03-11 06:30:00,433 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [3c4ee664] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 06:30:00,434 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-55) [6d4b6233] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:30:00,434 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-55) [6d4b6233] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:30 AM >2013-03-11 06:30:00,437 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-55) [6d4b6233] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:30 AM >2013-03-11 06:30:01,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [56ead237] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 67089cf2 >2013-03-11 06:30:01,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [56ead237] FINISH, GetHardwareInfoVDSCommand, log id: 67089cf2 >2013-03-11 06:30:01,692 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-61) [37f806f4] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 06:30:01,695 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-61) [7de6dee4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 06:30:01,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1e79ae43 >2013-03-11 06:30:08,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 06:30:08,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:30:08,740 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 06:30:08,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 1e79ae43 >2013-03-11 06:30:08,744 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [6e9ce048] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:30:08,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6e9ce048] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 21c7aae7 >2013-03-11 06:30:08,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [6e9ce048] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 06:30:08,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6e9ce048] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 21c7aae7 >2013-03-11 06:30:08,760 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [6e9ce048] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 06:30:08,760 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [7de6dee4] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 06:30:08,777 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-61) [25f7b2e4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:30:08,778 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [25f7b2e4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 43a2edf2 >2013-03-11 06:30:08,780 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [25f7b2e4] FINISH, SetVdsStatusVDSCommand, log id: 43a2edf2 >2013-03-11 06:30:08,788 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-61) [25f7b2e4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 06:30:08,811 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-61) [4035f26f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:30:08,813 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-61) [37f244d0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:30:08,815 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-61) [37f244d0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 06:30:33,873 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-41) [2b0dfecd] Start refreshing all users data >2013-03-11 06:30:35,060 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [5cd38640] Attempting to update VMs/Templates Ovf. >2013-03-11 06:30:35,061 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [5cd38640] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 06:30:35,061 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [5cd38640] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 06:30:35,062 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [5cd38640] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 06:30:35,062 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [5cd38640] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 06:30:35,062 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [5cd38640] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 06:30:35,063 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-43) [5cd38640] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 06:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [521f7032] Autorecovering 1 hosts >2013-03-11 06:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [521f7032] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [896d341] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [896d341] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [896d341] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6afcb305 >2013-03-11 06:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [896d341] FINISH, SetVdsStatusVDSCommand, log id: 6afcb305 >2013-03-11 06:35:00,018 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [896d341] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 59388602 >2013-03-11 06:35:00,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [896d341] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 48597baf >2013-03-11 06:35:00,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [896d341] FINISH, GetHardwareInfoVDSCommand, log id: 48597baf >2013-03-11 06:35:00,264 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-96) [165ca8b6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:35:00,266 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-96) [63883ed7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:35:00,268 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [63883ed7] FINISH, ActivateVdsVDSCommand, log id: 59388602 >2013-03-11 06:35:00,287 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [63883ed7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:35:00,288 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-96) [63883ed7] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 06:35:00,289 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [63883ed7] Autorecovering 1 storage domains >2013-03-11 06:35:00,289 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [63883ed7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 06:35:00,290 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [7cc89fa6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:35:00,290 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [7cc89fa6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:35 AM >2013-03-11 06:35:00,292 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [7cc89fa6] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:35 AM >2013-03-11 06:35:03,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [7199e66f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7ac9e03 >2013-03-11 06:35:03,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [7199e66f] FINISH, GetHardwareInfoVDSCommand, log id: 7ac9e03 >2013-03-11 06:35:03,092 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-75) [68ce3c89] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 06:35:03,096 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-75) [516efa7d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 06:35:03,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 622f8ed9 >2013-03-11 06:35:09,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 06:35:09,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:35:09,084 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 06:35:09,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 622f8ed9 >2013-03-11 06:35:09,087 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [71e2a858] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:35:09,101 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [71e2a858] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1d205d0a >2013-03-11 06:35:09,101 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [71e2a858] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 06:35:09,101 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [71e2a858] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1d205d0a >2013-03-11 06:35:09,103 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [71e2a858] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 06:35:09,103 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [516efa7d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 06:35:09,113 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-75) [366696e2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:35:09,114 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [366696e2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 16b340c7 >2013-03-11 06:35:09,115 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [366696e2] FINISH, SetVdsStatusVDSCommand, log id: 16b340c7 >2013-03-11 06:35:09,121 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-75) [366696e2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 06:35:09,144 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [3d115171] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:35:09,146 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [1322d614] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:35:09,148 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-75) [1322d614] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 06:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [31f5325a] Autorecovering 1 hosts >2013-03-11 06:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [31f5325a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [2e0aa2fe] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [2e0aa2fe] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [2e0aa2fe] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1be640c1 >2013-03-11 06:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [2e0aa2fe] FINISH, SetVdsStatusVDSCommand, log id: 1be640c1 >2013-03-11 06:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [2e0aa2fe] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2119ad3c >2013-03-11 06:40:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [2e0aa2fe] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7be91267 >2013-03-11 06:40:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [2e0aa2fe] FINISH, GetHardwareInfoVDSCommand, log id: 7be91267 >2013-03-11 06:40:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-73) [7056a222] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:40:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-73) [3b97b98f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:40:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [3b97b98f] FINISH, ActivateVdsVDSCommand, log id: 2119ad3c >2013-03-11 06:40:00,219 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [3b97b98f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:40:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-73) [3b97b98f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 06:40:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [3b97b98f] Autorecovering 1 storage domains >2013-03-11 06:40:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [3b97b98f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 06:40:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [7c26539f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:40:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [7c26539f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:40 AM >2013-03-11 06:40:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [7c26539f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:40 AM >2013-03-11 06:40:00,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [5904654d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4823c52a >2013-03-11 06:40:00,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [5904654d] FINISH, GetHardwareInfoVDSCommand, log id: 4823c52a >2013-03-11 06:40:00,459 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-59) [55e77dca] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 06:40:00,462 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-59) [5c5787b8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 06:40:00,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 72272341 >2013-03-11 06:40:07,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 06:40:07,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:40:07,455 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 06:40:07,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 72272341 >2013-03-11 06:40:07,459 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [18e0b328] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:40:07,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [18e0b328] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 25a8a011 >2013-03-11 06:40:07,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [18e0b328] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 06:40:07,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [18e0b328] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 25a8a011 >2013-03-11 06:40:07,477 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [5c5787b8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 06:40:07,486 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-59) [44ebc539] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:40:07,488 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [44ebc539] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 542b454e >2013-03-11 06:40:07,489 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [44ebc539] FINISH, SetVdsStatusVDSCommand, log id: 542b454e >2013-03-11 06:40:07,495 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [44ebc539] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 06:40:07,517 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [62230c51] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:40:07,520 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [336806d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:40:07,521 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-59) [336806d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 06:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [19ab7d79] Autorecovering 1 hosts >2013-03-11 06:45:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [19ab7d79] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [4780019e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [4780019e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [4780019e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 262f6893 >2013-03-11 06:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [4780019e] FINISH, SetVdsStatusVDSCommand, log id: 262f6893 >2013-03-11 06:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-1) [4780019e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 118e9eed >2013-03-11 06:45:00,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [4780019e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3d01252a >2013-03-11 06:45:00,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [4780019e] FINISH, GetHardwareInfoVDSCommand, log id: 3d01252a >2013-03-11 06:45:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-1) [7b30bc2a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:45:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-1) [fd43965] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:45:00,203 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-1) [fd43965] FINISH, ActivateVdsVDSCommand, log id: 118e9eed >2013-03-11 06:45:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [fd43965] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:45:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-1) [fd43965] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 06:45:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [fd43965] Autorecovering 1 storage domains >2013-03-11 06:45:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [fd43965] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 06:45:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [f7e3e4e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:45:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [f7e3e4e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:45 AM >2013-03-11 06:45:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [f7e3e4e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:45 AM >2013-03-11 06:45:01,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [4fb169fa] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7ffb6f0a >2013-03-11 06:45:01,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [4fb169fa] FINISH, GetHardwareInfoVDSCommand, log id: 7ffb6f0a >2013-03-11 06:45:01,801 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-21) [6c907a4d] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 06:45:01,804 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-21) [483425e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 06:45:01,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3c4763f2 >2013-03-11 06:45:06,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 06:45:06,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:45:06,793 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 06:45:06,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 3c4763f2 >2013-03-11 06:45:06,797 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [44db5c12] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:45:06,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [44db5c12] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 536fb54a >2013-03-11 06:45:06,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [44db5c12] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 06:45:06,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [44db5c12] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 536fb54a >2013-03-11 06:45:06,813 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [44db5c12] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 06:45:06,813 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [483425e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 06:45:06,822 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-21) [5a626b5b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:45:06,824 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [5a626b5b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2ed2e187 >2013-03-11 06:45:06,825 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [5a626b5b] FINISH, SetVdsStatusVDSCommand, log id: 2ed2e187 >2013-03-11 06:45:06,831 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-21) [5a626b5b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 06:45:06,853 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-21) [27d2888c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:45:06,856 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-21) [57edfeae] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:45:06,857 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-21) [57edfeae] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 06:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [68ad8fa8] Autorecovering 1 hosts >2013-03-11 06:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [68ad8fa8] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [168126ff] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [168126ff] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:50:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [168126ff] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 496369a0 >2013-03-11 06:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [168126ff] FINISH, SetVdsStatusVDSCommand, log id: 496369a0 >2013-03-11 06:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [168126ff] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 52435d92 >2013-03-11 06:50:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [168126ff] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 69ffbc12 >2013-03-11 06:50:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [168126ff] FINISH, GetHardwareInfoVDSCommand, log id: 69ffbc12 >2013-03-11 06:50:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-14) [3f314620] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:50:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-14) [45d46dc9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:50:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [45d46dc9] FINISH, ActivateVdsVDSCommand, log id: 52435d92 >2013-03-11 06:50:00,223 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [45d46dc9] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:50:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-14) [45d46dc9] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 06:50:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [45d46dc9] Autorecovering 1 storage domains >2013-03-11 06:50:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [45d46dc9] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 06:50:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [1b5a85ea] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:50:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [1b5a85ea] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:50 AM >2013-03-11 06:50:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [1b5a85ea] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:50 AM >2013-03-11 06:50:01,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-30) [5cb2d43f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2f05fa68 >2013-03-11 06:50:01,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-30) [5cb2d43f] FINISH, GetHardwareInfoVDSCommand, log id: 2f05fa68 >2013-03-11 06:50:01,142 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-30) [56e52d0c] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 06:50:01,145 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-30) [77b279bc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 06:50:01,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7f2f817f >2013-03-11 06:50:07,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 06:50:07,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:50:07,130 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 06:50:07,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 7f2f817f >2013-03-11 06:50:07,133 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [3578d67c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:50:07,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [3578d67c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 70494318 >2013-03-11 06:50:07,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [3578d67c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 06:50:07,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [3578d67c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 70494318 >2013-03-11 06:50:07,149 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [3578d67c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 06:50:07,149 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [77b279bc] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 06:50:07,158 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-30) [1f6e229a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:50:07,160 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-30) [1f6e229a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 65c2065b >2013-03-11 06:50:07,161 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-30) [1f6e229a] FINISH, SetVdsStatusVDSCommand, log id: 65c2065b >2013-03-11 06:50:07,166 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-30) [1f6e229a] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 06:50:07,189 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-30) [19554379] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:50:07,191 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-30) [1874f45f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:50:07,193 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-30) [1874f45f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 06:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [7c26539f] Autorecovering 1 hosts >2013-03-11 06:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [7c26539f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [7708d7a0] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [7708d7a0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [7708d7a0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1559209d >2013-03-11 06:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [7708d7a0] FINISH, SetVdsStatusVDSCommand, log id: 1559209d >2013-03-11 06:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [7708d7a0] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 51c1a793 >2013-03-11 06:55:00,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [7708d7a0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7f5fbc2a >2013-03-11 06:55:00,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [7708d7a0] FINISH, GetHardwareInfoVDSCommand, log id: 7f5fbc2a >2013-03-11 06:55:00,584 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-73) [449f8df4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:55:00,587 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-73) [7c7d5425] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:55:00,589 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [7c7d5425] FINISH, ActivateVdsVDSCommand, log id: 51c1a793 >2013-03-11 06:55:00,608 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [7c7d5425] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 06:55:00,609 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-73) [7c7d5425] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 06:55:00,610 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [7c7d5425] Autorecovering 1 storage domains >2013-03-11 06:55:00,610 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [7c7d5425] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 06:55:00,611 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [542eb785] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:55:00,611 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [542eb785] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:55 AM >2013-03-11 06:55:00,614 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [542eb785] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:55 AM >2013-03-11 06:55:01,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [524c4662] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2158670b >2013-03-11 06:55:01,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [524c4662] FINISH, GetHardwareInfoVDSCommand, log id: 2158670b >2013-03-11 06:55:01,486 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-36) [229fc76f] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 06:55:01,490 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-36) [3ce5f64c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 06:55:01,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 57a8a209 >2013-03-11 06:55:07,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 06:55:07,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 06:55:07,524 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 06:55:07,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 57a8a209 >2013-03-11 06:55:07,527 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [331cd209] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 06:55:07,541 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [331cd209] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 18bbc697 >2013-03-11 06:55:07,541 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [331cd209] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 06:55:07,541 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [331cd209] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 18bbc697 >2013-03-11 06:55:07,543 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [331cd209] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 06:55:07,543 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [3ce5f64c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 06:55:07,553 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-36) [4cc3763e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:55:07,555 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [4cc3763e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 384311fd >2013-03-11 06:55:07,556 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [4cc3763e] FINISH, SetVdsStatusVDSCommand, log id: 384311fd >2013-03-11 06:55:07,562 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [4cc3763e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 06:55:07,585 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [5c0b7dac] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:55:07,588 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [56ad64a3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 06:55:07,589 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-36) [56ad64a3] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 07:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [1bc1025e] Autorecovering 1 hosts >2013-03-11 07:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [1bc1025e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [504fddd] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [504fddd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [504fddd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7f88ce46 >2013-03-11 07:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [504fddd] FINISH, SetVdsStatusVDSCommand, log id: 7f88ce46 >2013-03-11 07:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [504fddd] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6de5391 >2013-03-11 07:00:00,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [504fddd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 53ec79b4 >2013-03-11 07:00:00,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [504fddd] FINISH, GetHardwareInfoVDSCommand, log id: 53ec79b4 >2013-03-11 07:00:00,098 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [6484d60e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:00:00,100 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [33491015] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:00:00,101 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [33491015] FINISH, ActivateVdsVDSCommand, log id: 6de5391 >2013-03-11 07:00:00,121 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [33491015] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:00:00,122 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [33491015] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 07:00:00,123 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [33491015] Autorecovering 1 storage domains >2013-03-11 07:00:00,124 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [33491015] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 07:00:00,125 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [74eede3d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:00:00,125 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [74eede3d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 7:00 AM >2013-03-11 07:00:00,128 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [74eede3d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 7:00 AM >2013-03-11 07:00:01,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [f7e3e4e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2924015d >2013-03-11 07:00:01,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [f7e3e4e] FINISH, GetHardwareInfoVDSCommand, log id: 2924015d >2013-03-11 07:00:01,870 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-1) [71c984eb] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 07:00:01,874 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-1) [1732cb91] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 07:00:01,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 75e5d4ab >2013-03-11 07:00:08,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 07:00:08,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:00:08,943 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 07:00:08,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 75e5d4ab >2013-03-11 07:00:08,946 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [174786e0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:00:08,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [174786e0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c083a55 >2013-03-11 07:00:08,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [174786e0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 07:00:08,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [174786e0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c083a55 >2013-03-11 07:00:08,963 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [174786e0] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 07:00:08,964 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [1732cb91] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 07:00:08,973 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-1) [ebab064] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:00:08,974 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [ebab064] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3dfd7ce8 >2013-03-11 07:00:08,975 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [ebab064] FINISH, SetVdsStatusVDSCommand, log id: 3dfd7ce8 >2013-03-11 07:00:08,981 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-1) [ebab064] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 07:00:09,006 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-1) [4e65e964] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:00:09,009 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-1) [5e033ff9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:00:09,010 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-1) [5e033ff9] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 07:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [45627a46] Autorecovering 1 hosts >2013-03-11 07:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [45627a46] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-95) [6f5293b1] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-95) [6f5293b1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [6f5293b1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 435a96ca >2013-03-11 07:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [6f5293b1] FINISH, SetVdsStatusVDSCommand, log id: 435a96ca >2013-03-11 07:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-95) [6f5293b1] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 65f49f1f >2013-03-11 07:05:00,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [6f5293b1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1b45ca81 >2013-03-11 07:05:00,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [6f5293b1] FINISH, GetHardwareInfoVDSCommand, log id: 1b45ca81 >2013-03-11 07:05:00,345 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-95) [14ce185a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:05:00,348 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-95) [257b52c4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:05:00,349 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-95) [257b52c4] FINISH, ActivateVdsVDSCommand, log id: 65f49f1f >2013-03-11 07:05:00,368 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-95) [257b52c4] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:05:00,369 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-95) [257b52c4] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 07:05:00,371 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [257b52c4] Autorecovering 1 storage domains >2013-03-11 07:05:00,371 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [257b52c4] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 07:05:00,371 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-95) [eafeceb] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:05:00,371 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-95) [eafeceb] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 7:05 AM >2013-03-11 07:05:00,374 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-95) [eafeceb] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 7:05 AM >2013-03-11 07:05:00,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [6dc1a565] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4936268a >2013-03-11 07:05:00,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [6dc1a565] FINISH, GetHardwareInfoVDSCommand, log id: 4936268a >2013-03-11 07:05:00,447 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-5) [34702f9e] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 07:05:00,450 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-5) [6a2b8ace] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 07:05:00,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 309097d7 >2013-03-11 07:05:05,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 07:05:05,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:05:05,449 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 07:05:05,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 309097d7 >2013-03-11 07:05:05,452 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [72499605] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:05:05,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [72499605] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40b2bd09 >2013-03-11 07:05:05,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [72499605] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 07:05:05,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [72499605] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40b2bd09 >2013-03-11 07:05:05,468 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [72499605] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 07:05:05,468 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [6a2b8ace] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 07:05:05,477 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-5) [35a9a5a5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:05:05,479 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [35a9a5a5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 36c6376d >2013-03-11 07:05:05,480 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [35a9a5a5] FINISH, SetVdsStatusVDSCommand, log id: 36c6376d >2013-03-11 07:05:05,486 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-5) [35a9a5a5] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 07:05:05,527 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-5) [2cc740] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:05:05,530 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-5) [7991bad5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:05:05,531 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-5) [7991bad5] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 07:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [7a32b6f5] Autorecovering 1 hosts >2013-03-11 07:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [7a32b6f5] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [4aba9114] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [4aba9114] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:10:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [4aba9114] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3d977904 >2013-03-11 07:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [4aba9114] FINISH, SetVdsStatusVDSCommand, log id: 3d977904 >2013-03-11 07:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [4aba9114] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 34d674a5 >2013-03-11 07:10:00,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [4aba9114] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 74a70fb7 >2013-03-11 07:10:00,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [4aba9114] FINISH, GetHardwareInfoVDSCommand, log id: 74a70fb7 >2013-03-11 07:10:00,330 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [207be12e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:10:00,332 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [cc7420e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:10:00,333 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-94) [cc7420e] FINISH, ActivateVdsVDSCommand, log id: 34d674a5 >2013-03-11 07:10:00,352 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-94) [cc7420e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:10:00,353 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-94) [cc7420e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 07:10:00,355 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [cc7420e] Autorecovering 1 storage domains >2013-03-11 07:10:00,355 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-94) [cc7420e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 07:10:00,355 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [452f63bc] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:10:00,355 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [452f63bc] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 7:10 AM >2013-03-11 07:10:00,358 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-94) [452f63bc] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 7:10 AM >2013-03-11 07:10:02,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [542eb785] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4889a47a >2013-03-11 07:10:02,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [542eb785] FINISH, GetHardwareInfoVDSCommand, log id: 4889a47a >2013-03-11 07:10:02,810 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-73) [5d7409a5] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 07:10:02,814 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-73) [a3d5e8a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 07:10:02,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5737355b >2013-03-11 07:10:08,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 07:10:08,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:10:08,818 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 07:10:08,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 5737355b >2013-03-11 07:10:08,822 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [699f157] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:10:08,836 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [699f157] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 43e395c3 >2013-03-11 07:10:08,836 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [699f157] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 07:10:08,836 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [699f157] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 43e395c3 >2013-03-11 07:10:08,838 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [699f157] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 07:10:08,838 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [a3d5e8a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 07:10:08,847 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-73) [53686648] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:10:08,849 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [53686648] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 453146bb >2013-03-11 07:10:08,850 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [53686648] FINISH, SetVdsStatusVDSCommand, log id: 453146bb >2013-03-11 07:10:08,856 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-73) [53686648] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 07:10:08,879 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-73) [64414e4b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:10:08,881 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-73) [5e18beab] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:10:08,882 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-73) [5e18beab] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 07:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [41ad3119] Autorecovering 1 hosts >2013-03-11 07:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [41ad3119] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [2d458e49] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [2d458e49] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [2d458e49] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 199f20b4 >2013-03-11 07:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [2d458e49] FINISH, SetVdsStatusVDSCommand, log id: 199f20b4 >2013-03-11 07:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-70) [2d458e49] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 24df1cf5 >2013-03-11 07:15:00,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [2d458e49] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 70899c0 >2013-03-11 07:15:00,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [2d458e49] FINISH, GetHardwareInfoVDSCommand, log id: 70899c0 >2013-03-11 07:15:00,295 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-70) [342e6264] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:15:00,330 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-70) [18e6c27c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:15:00,332 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-70) [18e6c27c] FINISH, ActivateVdsVDSCommand, log id: 24df1cf5 >2013-03-11 07:15:00,358 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [18e6c27c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:15:00,359 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-70) [18e6c27c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 07:15:00,360 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [18e6c27c] Autorecovering 1 storage domains >2013-03-11 07:15:00,360 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [18e6c27c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 07:15:00,361 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [6206e7e8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:15:00,361 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [6206e7e8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 7:15 AM >2013-03-11 07:15:00,364 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [6206e7e8] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 7:15 AM >2013-03-11 07:15:00,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [74eede3d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 49b803b3 >2013-03-11 07:15:00,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [74eede3d] FINISH, GetHardwareInfoVDSCommand, log id: 49b803b3 >2013-03-11 07:15:00,429 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-26) [554e7fae] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 07:15:00,432 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-26) [4a17f947] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 07:15:00,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 565a4733 >2013-03-11 07:15:05,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 07:15:05,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:15:05,428 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 07:15:05,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 565a4733 >2013-03-11 07:15:05,431 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [72184ffd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:15:05,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [72184ffd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3b707f37 >2013-03-11 07:15:05,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [72184ffd] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 07:15:05,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [72184ffd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3b707f37 >2013-03-11 07:15:05,447 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [72184ffd] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 07:15:05,447 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [4a17f947] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 07:15:05,457 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-26) [b0f47af] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:15:05,458 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [b0f47af] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 77dc9f44 >2013-03-11 07:15:05,460 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [b0f47af] FINISH, SetVdsStatusVDSCommand, log id: 77dc9f44 >2013-03-11 07:15:05,465 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [b0f47af] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 07:15:05,488 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [55067d17] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:15:05,491 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [572158f7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:15:05,492 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-26) [572158f7] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 07:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [f1e7b08] Autorecovering 1 hosts >2013-03-11 07:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [f1e7b08] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [ab04626] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [ab04626] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [ab04626] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7239a7d7 >2013-03-11 07:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [ab04626] FINISH, SetVdsStatusVDSCommand, log id: 7239a7d7 >2013-03-11 07:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-2) [ab04626] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 79b90d26 >2013-03-11 07:20:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [ab04626] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5287d0d2 >2013-03-11 07:20:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [ab04626] FINISH, GetHardwareInfoVDSCommand, log id: 5287d0d2 >2013-03-11 07:20:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [42e0db05] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:20:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [30d8bd08] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:20:00,203 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-2) [30d8bd08] FINISH, ActivateVdsVDSCommand, log id: 79b90d26 >2013-03-11 07:20:00,222 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [30d8bd08] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:20:00,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [30d8bd08] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 07:20:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [30d8bd08] Autorecovering 1 storage domains >2013-03-11 07:20:00,225 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [30d8bd08] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 07:20:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [72afb593] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:20:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [72afb593] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 7:20 AM >2013-03-11 07:20:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [72afb593] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 7:20 AM >2013-03-11 07:20:02,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [1b5a85ea] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 553b7f76 >2013-03-11 07:20:02,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [1b5a85ea] FINISH, GetHardwareInfoVDSCommand, log id: 553b7f76 >2013-03-11 07:20:02,777 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-14) [571344e0] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 07:20:02,780 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-14) [19442fec] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 07:20:02,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2613af82 >2013-03-11 07:20:08,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 07:20:08,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:20:08,783 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 07:20:08,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2613af82 >2013-03-11 07:20:08,787 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [2d125075] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:20:08,809 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [2d125075] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 91bed75 >2013-03-11 07:20:08,809 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [2d125075] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 07:20:08,809 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [2d125075] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 91bed75 >2013-03-11 07:20:08,811 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [2d125075] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 07:20:08,811 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [19442fec] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 07:20:08,821 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-14) [51473767] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:20:08,823 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [51473767] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 701f7559 >2013-03-11 07:20:08,824 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [51473767] FINISH, SetVdsStatusVDSCommand, log id: 701f7559 >2013-03-11 07:20:08,830 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-14) [51473767] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 07:20:08,853 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-14) [20c12560] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:20:08,856 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-14) [33d29aab] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:20:08,857 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-14) [33d29aab] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 07:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [7c928d83] Autorecovering 1 hosts >2013-03-11 07:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [7c928d83] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [68a4b8b8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [68a4b8b8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:25:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [68a4b8b8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6851d7a8 >2013-03-11 07:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [68a4b8b8] FINISH, SetVdsStatusVDSCommand, log id: 6851d7a8 >2013-03-11 07:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-33) [68a4b8b8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 64d1fe9a >2013-03-11 07:25:00,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [68a4b8b8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4e835fd0 >2013-03-11 07:25:00,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [68a4b8b8] FINISH, GetHardwareInfoVDSCommand, log id: 4e835fd0 >2013-03-11 07:25:00,403 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-33) [216660f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:25:00,406 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-33) [c41995b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:25:00,407 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-33) [c41995b] FINISH, ActivateVdsVDSCommand, log id: 64d1fe9a >2013-03-11 07:25:00,426 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [c41995b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:25:00,427 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-33) [c41995b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 07:25:00,428 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [c41995b] Autorecovering 1 storage domains >2013-03-11 07:25:00,428 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [c41995b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 07:25:00,429 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [4fb2396] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:25:00,429 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [4fb2396] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 7:25 AM >2013-03-11 07:25:00,432 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [4fb2396] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 7:25 AM >2013-03-11 07:25:00,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [6110a170] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7cbb4c6a >2013-03-11 07:25:00,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [6110a170] FINISH, GetHardwareInfoVDSCommand, log id: 7cbb4c6a >2013-03-11 07:25:00,504 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-18) [74f680db] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 07:25:00,507 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-18) [2c0d9574] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 07:25:00,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 60abb2d5 >2013-03-11 07:25:06,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 07:25:06,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:25:06,503 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 07:25:06,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 60abb2d5 >2013-03-11 07:25:06,506 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [15cd0375] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:25:06,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [15cd0375] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7337ace0 >2013-03-11 07:25:06,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [15cd0375] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 07:25:06,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [15cd0375] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7337ace0 >2013-03-11 07:25:06,522 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [15cd0375] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 07:25:06,523 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [2c0d9574] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 07:25:06,532 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-18) [36fcef8a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:25:06,534 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [36fcef8a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 67543b89 >2013-03-11 07:25:06,535 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [36fcef8a] FINISH, SetVdsStatusVDSCommand, log id: 67543b89 >2013-03-11 07:25:06,542 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [36fcef8a] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 07:25:06,564 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [4f4a6de2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:25:06,567 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [493a1ca4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:25:06,568 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) [493a1ca4] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 07:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [503f561c] Autorecovering 1 hosts >2013-03-11 07:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [503f561c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-49) [3e5b3137] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-49) [3e5b3137] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [3e5b3137] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1a10863c >2013-03-11 07:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [3e5b3137] FINISH, SetVdsStatusVDSCommand, log id: 1a10863c >2013-03-11 07:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-49) [3e5b3137] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7468ea0 >2013-03-11 07:30:00,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [3e5b3137] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7ba37fe5 >2013-03-11 07:30:00,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [3e5b3137] FINISH, GetHardwareInfoVDSCommand, log id: 7ba37fe5 >2013-03-11 07:30:00,262 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-49) [4d2aaf87] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:30:00,264 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-49) [1f87bf02] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:30:00,265 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-49) [1f87bf02] FINISH, ActivateVdsVDSCommand, log id: 7468ea0 >2013-03-11 07:30:00,285 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-49) [1f87bf02] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:30:00,285 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-49) [1f87bf02] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 07:30:00,287 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [1f87bf02] Autorecovering 1 storage domains >2013-03-11 07:30:00,287 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [1f87bf02] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 07:30:00,287 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-49) [70f9810a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:30:00,288 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-49) [70f9810a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 7:30 AM >2013-03-11 07:30:00,290 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-49) [70f9810a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 7:30 AM >2013-03-11 07:30:00,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [2733029b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 78727098 >2013-03-11 07:30:00,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [2733029b] FINISH, GetHardwareInfoVDSCommand, log id: 78727098 >2013-03-11 07:30:00,855 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-45) [21c74b2f] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 07:30:00,858 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-45) [30c35f54] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 07:30:00,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4ab53caa >2013-03-11 07:30:07,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 07:30:07,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:30:07,910 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 07:30:07,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 4ab53caa >2013-03-11 07:30:07,914 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [39c07322] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:30:07,927 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [39c07322] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5b9ad304 >2013-03-11 07:30:07,927 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [39c07322] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 07:30:07,927 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [39c07322] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5b9ad304 >2013-03-11 07:30:07,929 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [39c07322] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 07:30:07,930 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [30c35f54] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 07:30:07,938 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-45) [62004a15] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:30:07,940 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [62004a15] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 668509a1 >2013-03-11 07:30:07,941 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [62004a15] FINISH, SetVdsStatusVDSCommand, log id: 668509a1 >2013-03-11 07:30:07,947 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-45) [62004a15] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 07:30:07,970 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-45) [4a57e16] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:30:07,972 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-45) [1410c527] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:30:07,974 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-45) [1410c527] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 07:30:33,875 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-24) [137d1c0a] Start refreshing all users data >2013-03-11 07:30:35,064 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) [56ad64a3] Attempting to update VMs/Templates Ovf. >2013-03-11 07:30:35,065 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) [56ad64a3] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 07:30:35,065 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) [56ad64a3] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 07:30:35,065 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) [56ad64a3] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 07:30:35,066 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) [56ad64a3] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 07:30:35,066 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) [56ad64a3] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 07:30:35,067 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-36) [56ad64a3] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 07:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [72afb593] Autorecovering 1 hosts >2013-03-11 07:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [72afb593] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [772ba312] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [772ba312] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [772ba312] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 28b964e >2013-03-11 07:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [772ba312] FINISH, SetVdsStatusVDSCommand, log id: 28b964e >2013-03-11 07:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-2) [772ba312] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 79d096f6 >2013-03-11 07:35:00,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [772ba312] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5cce11ff >2013-03-11 07:35:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [772ba312] FINISH, GetHardwareInfoVDSCommand, log id: 5cce11ff >2013-03-11 07:35:00,183 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [fe7f19a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:35:00,185 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [cfb0084] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:35:00,187 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-2) [cfb0084] FINISH, ActivateVdsVDSCommand, log id: 79d096f6 >2013-03-11 07:35:00,205 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [cfb0084] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:35:00,206 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [cfb0084] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 07:35:00,208 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [cfb0084] Autorecovering 1 storage domains >2013-03-11 07:35:00,208 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [cfb0084] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 07:35:00,208 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [d05bffd] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:35:00,208 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [d05bffd] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 7:35 AM >2013-03-11 07:35:00,211 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [d05bffd] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 7:35 AM >2013-03-11 07:35:02,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [8f8b439] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 90c7718 >2013-03-11 07:35:02,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [8f8b439] FINISH, GetHardwareInfoVDSCommand, log id: 90c7718 >2013-03-11 07:35:02,252 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-58) [1c4742e0] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 07:35:02,256 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-58) [2ce7d15f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 07:35:02,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4c16744 >2013-03-11 07:35:07,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 07:35:07,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:35:07,253 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 07:35:07,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 4c16744 >2013-03-11 07:35:07,257 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [4977449d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:35:07,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4977449d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 740526c0 >2013-03-11 07:35:07,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [4977449d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 07:35:07,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4977449d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 740526c0 >2013-03-11 07:35:07,273 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [4977449d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 07:35:07,273 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [2ce7d15f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 07:35:07,282 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-58) [4df5b3e3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:35:07,284 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [4df5b3e3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 43c071d >2013-03-11 07:35:07,285 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [4df5b3e3] FINISH, SetVdsStatusVDSCommand, log id: 43c071d >2013-03-11 07:35:07,291 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-58) [4df5b3e3] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 07:35:07,315 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-58) [15a4cb14] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:35:07,318 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-58) [4ed0941e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:35:07,319 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-58) [4ed0941e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 07:40:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [4fb2396] Autorecovering 1 hosts >2013-03-11 07:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [4fb2396] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [7be04926] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [7be04926] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:40:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [7be04926] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4f20ea68 >2013-03-11 07:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [7be04926] FINISH, SetVdsStatusVDSCommand, log id: 4f20ea68 >2013-03-11 07:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-33) [7be04926] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1baf726e >2013-03-11 07:40:00,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [7be04926] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 77928a64 >2013-03-11 07:40:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [7be04926] FINISH, GetHardwareInfoVDSCommand, log id: 77928a64 >2013-03-11 07:40:00,183 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-33) [2b69a20f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:40:00,185 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-33) [61f0fea] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:40:00,186 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-33) [61f0fea] FINISH, ActivateVdsVDSCommand, log id: 1baf726e >2013-03-11 07:40:00,206 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [61f0fea] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:40:00,206 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-33) [61f0fea] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 07:40:00,208 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [61f0fea] Autorecovering 1 storage domains >2013-03-11 07:40:00,208 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [61f0fea] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 07:40:00,208 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [5a1cb509] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:40:00,209 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [5a1cb509] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 7:40 AM >2013-03-11 07:40:00,211 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [5a1cb509] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 7:40 AM >2013-03-11 07:40:01,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-71) [57ef1896] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 400519a8 >2013-03-11 07:40:01,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-71) [57ef1896] FINISH, GetHardwareInfoVDSCommand, log id: 400519a8 >2013-03-11 07:40:01,594 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-71) [783588cf] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 07:40:01,598 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-71) [1828d4de] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 07:40:01,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2ff67727 >2013-03-11 07:40:05,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 07:40:05,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:40:05,595 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 07:40:05,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 2ff67727 >2013-03-11 07:40:05,599 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [41fc26b5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:40:05,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [41fc26b5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7cf5f88b >2013-03-11 07:40:05,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [41fc26b5] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 07:40:05,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [41fc26b5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7cf5f88b >2013-03-11 07:40:05,621 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [41fc26b5] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 07:40:05,622 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [1828d4de] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 07:40:05,631 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-71) [2b1de38d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:40:05,632 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-71) [2b1de38d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 9e6a658 >2013-03-11 07:40:05,634 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-71) [2b1de38d] FINISH, SetVdsStatusVDSCommand, log id: 9e6a658 >2013-03-11 07:40:05,639 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-71) [2b1de38d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 07:40:05,662 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-71) [10e1551c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:40:05,665 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-71) [1c24a17d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:40:05,666 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-71) [1c24a17d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 07:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [6a82214e] Autorecovering 1 hosts >2013-03-11 07:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [6a82214e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [74105d57] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [74105d57] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [74105d57] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2bc04b0d >2013-03-11 07:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [74105d57] FINISH, SetVdsStatusVDSCommand, log id: 2bc04b0d >2013-03-11 07:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-27) [74105d57] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: a0a67c1 >2013-03-11 07:45:00,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [74105d57] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: eeb9793 >2013-03-11 07:45:00,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [74105d57] FINISH, GetHardwareInfoVDSCommand, log id: eeb9793 >2013-03-11 07:45:00,207 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-27) [3c1f243] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:45:00,209 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-27) [407b20a1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:45:00,211 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-27) [407b20a1] FINISH, ActivateVdsVDSCommand, log id: a0a67c1 >2013-03-11 07:45:00,230 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [407b20a1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:45:00,231 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-27) [407b20a1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 07:45:00,232 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [407b20a1] Autorecovering 1 storage domains >2013-03-11 07:45:00,232 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [407b20a1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 07:45:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [2164ac50] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:45:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [2164ac50] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 7:45 AM >2013-03-11 07:45:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [2164ac50] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 7:45 AM >2013-03-11 07:45:02,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [6d4b6233] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 26ecf6dd >2013-03-11 07:45:02,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [6d4b6233] FINISH, GetHardwareInfoVDSCommand, log id: 26ecf6dd >2013-03-11 07:45:02,950 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-55) [4432d237] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 07:45:02,954 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-55) [1ea34a37] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 07:45:02,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1d1113a1 >2013-03-11 07:45:07,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 07:45:07,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:45:07,939 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 07:45:07,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 1d1113a1 >2013-03-11 07:45:07,943 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [6833e737] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:45:07,956 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6833e737] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 146d24e9 >2013-03-11 07:45:07,956 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [6833e737] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 07:45:07,956 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6833e737] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 146d24e9 >2013-03-11 07:45:07,961 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [1ea34a37] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 07:45:07,970 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-55) [15f2148b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:45:07,972 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [15f2148b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 73324d9d >2013-03-11 07:45:07,973 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [15f2148b] FINISH, SetVdsStatusVDSCommand, log id: 73324d9d >2013-03-11 07:45:07,979 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-55) [15f2148b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 07:45:08,002 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-55) [2f69efc6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:45:08,005 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-55) [5590818f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:45:08,006 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-55) [5590818f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 07:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [6cc0757f] Autorecovering 1 hosts >2013-03-11 07:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [6cc0757f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [319e6916] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [319e6916] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [319e6916] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4652aafa >2013-03-11 07:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [319e6916] FINISH, SetVdsStatusVDSCommand, log id: 4652aafa >2013-03-11 07:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [319e6916] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 411b2599 >2013-03-11 07:50:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [319e6916] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 207cbcd3 >2013-03-11 07:50:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [319e6916] FINISH, GetHardwareInfoVDSCommand, log id: 207cbcd3 >2013-03-11 07:50:00,257 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-42) [202a4761] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:50:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-42) [1dc655de] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:50:00,261 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [1dc655de] FINISH, ActivateVdsVDSCommand, log id: 411b2599 >2013-03-11 07:50:00,281 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [1dc655de] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:50:00,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-42) [1dc655de] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 07:50:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [1dc655de] Autorecovering 1 storage domains >2013-03-11 07:50:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [1dc655de] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 07:50:00,283 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [7b4e7db5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:50:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [7b4e7db5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 7:50 AM >2013-03-11 07:50:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [7b4e7db5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 7:50 AM >2013-03-11 07:50:02,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [6a2184f9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 330e2b83 >2013-03-11 07:50:02,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [6a2184f9] FINISH, GetHardwareInfoVDSCommand, log id: 330e2b83 >2013-03-11 07:50:02,283 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-37) [428b89bc] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 07:50:02,286 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-37) [3f7a4467] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 07:50:02,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 15a1cefc >2013-03-11 07:50:06,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 07:50:06,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:50:06,875 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 07:50:06,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 15a1cefc >2013-03-11 07:50:06,878 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [21d7dfb0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:50:06,892 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [21d7dfb0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7bb37f4c >2013-03-11 07:50:06,892 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [21d7dfb0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 07:50:06,892 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [21d7dfb0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7bb37f4c >2013-03-11 07:50:06,894 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [21d7dfb0] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 07:50:06,895 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [3f7a4467] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 07:50:06,904 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-37) [7483e99b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:50:06,906 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [7483e99b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 226f73f1 >2013-03-11 07:50:06,907 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [7483e99b] FINISH, SetVdsStatusVDSCommand, log id: 226f73f1 >2013-03-11 07:50:06,913 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-37) [7483e99b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 07:50:06,936 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-37) [54fadcba] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:50:06,938 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-37) [12f4bb98] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:50:06,940 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-37) [12f4bb98] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 07:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [5a1cb509] Autorecovering 1 hosts >2013-03-11 07:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [5a1cb509] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [3e6de918] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [3e6de918] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [3e6de918] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3fa4f815 >2013-03-11 07:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [3e6de918] FINISH, SetVdsStatusVDSCommand, log id: 3fa4f815 >2013-03-11 07:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-33) [3e6de918] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6194b45f >2013-03-11 07:55:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [3e6de918] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 77480043 >2013-03-11 07:55:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [3e6de918] FINISH, GetHardwareInfoVDSCommand, log id: 77480043 >2013-03-11 07:55:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-33) [17eb3e5c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:55:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-33) [3cfb8b3e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:55:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-33) [3cfb8b3e] FINISH, ActivateVdsVDSCommand, log id: 6194b45f >2013-03-11 07:55:00,221 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-33) [3cfb8b3e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 07:55:00,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-33) [3cfb8b3e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 07:55:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [3cfb8b3e] Autorecovering 1 storage domains >2013-03-11 07:55:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-33) [3cfb8b3e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 07:55:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [5560d88a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:55:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [5560d88a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 7:55 AM >2013-03-11 07:55:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-33) [5560d88a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 7:55 AM >2013-03-11 07:55:01,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [493a1ca4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1d6d2e5e >2013-03-11 07:55:01,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [493a1ca4] FINISH, GetHardwareInfoVDSCommand, log id: 1d6d2e5e >2013-03-11 07:55:01,223 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-18) [4839add0] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 07:55:01,226 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-18) [58f977af] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 07:55:01,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: ac9e79b >2013-03-11 07:55:08,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 07:55:08,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 07:55:08,221 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 07:55:08,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: ac9e79b >2013-03-11 07:55:08,224 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [4a982528] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 07:55:08,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4a982528] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 415837a7 >2013-03-11 07:55:08,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [4a982528] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 07:55:08,238 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4a982528] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 415837a7 >2013-03-11 07:55:08,240 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [4a982528] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 07:55:08,241 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [58f977af] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 07:55:08,250 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-18) [7bc440f3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:55:08,252 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [7bc440f3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2e9dc6bf >2013-03-11 07:55:08,253 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [7bc440f3] FINISH, SetVdsStatusVDSCommand, log id: 2e9dc6bf >2013-03-11 07:55:08,258 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [7bc440f3] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 07:55:08,282 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [2efe5dc1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:55:08,285 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [5e21f447] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 07:55:08,286 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) [5e21f447] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 08:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [70f9810a] Autorecovering 1 hosts >2013-03-11 08:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [70f9810a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-49) [3db64196] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-49) [3db64196] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [3db64196] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 56a745f6 >2013-03-11 08:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [3db64196] FINISH, SetVdsStatusVDSCommand, log id: 56a745f6 >2013-03-11 08:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-49) [3db64196] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: e50bd2b >2013-03-11 08:00:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [3db64196] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 18efa096 >2013-03-11 08:00:00,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [3db64196] FINISH, GetHardwareInfoVDSCommand, log id: 18efa096 >2013-03-11 08:00:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-49) [6d2ac5be] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:00:00,206 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-49) [445212d9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:00:00,207 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-49) [445212d9] FINISH, ActivateVdsVDSCommand, log id: e50bd2b >2013-03-11 08:00:00,226 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-49) [445212d9] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:00:00,227 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-49) [445212d9] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 08:00:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [445212d9] Autorecovering 1 storage domains >2013-03-11 08:00:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [445212d9] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 08:00:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-49) [5c8317b9] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:00:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-49) [5c8317b9] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 8:00 AM >2013-03-11 08:00:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-49) [5c8317b9] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 8:00 AM >2013-03-11 08:00:02,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [2164ac50] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 13d103a5 >2013-03-11 08:00:02,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [2164ac50] FINISH, GetHardwareInfoVDSCommand, log id: 13d103a5 >2013-03-11 08:00:02,564 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-27) [4381c67b] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 08:00:02,568 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-27) [1f7c70ad] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 08:00:02,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2e195648 >2013-03-11 08:00:08,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 08:00:08,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:00:08,611 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 08:00:08,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 2e195648 >2013-03-11 08:00:08,615 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [27bcf5d3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:00:08,628 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [27bcf5d3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 41262802 >2013-03-11 08:00:08,628 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [27bcf5d3] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 08:00:08,628 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [27bcf5d3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 41262802 >2013-03-11 08:00:08,630 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [27bcf5d3] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 08:00:08,631 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [1f7c70ad] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 08:00:08,640 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-27) [7649f2bd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:00:08,642 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [7649f2bd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 112c809a >2013-03-11 08:00:08,643 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [7649f2bd] FINISH, SetVdsStatusVDSCommand, log id: 112c809a >2013-03-11 08:00:08,649 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-27) [7649f2bd] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 08:00:08,671 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-27) [68d0c3ad] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:00:08,673 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-27) [1bfec205] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:00:08,675 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-27) [1bfec205] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 08:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [29b43c6b] Autorecovering 1 hosts >2013-03-11 08:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [29b43c6b] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [30966133] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [30966133] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [30966133] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 25f57bb3 >2013-03-11 08:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [30966133] FINISH, SetVdsStatusVDSCommand, log id: 25f57bb3 >2013-03-11 08:05:00,010 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-93) [30966133] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 401b1df1 >2013-03-11 08:05:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [30966133] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5432d13 >2013-03-11 08:05:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [30966133] FINISH, GetHardwareInfoVDSCommand, log id: 5432d13 >2013-03-11 08:05:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [7a9b23b4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:05:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [49c85906] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:05:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-93) [49c85906] FINISH, ActivateVdsVDSCommand, log id: 401b1df1 >2013-03-11 08:05:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [49c85906] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:05:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-93) [49c85906] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 08:05:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [49c85906] Autorecovering 1 storage domains >2013-03-11 08:05:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [49c85906] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 08:05:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [6dd1154] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:05:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [6dd1154] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 8:05 AM >2013-03-11 08:05:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [6dd1154] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 8:05 AM >2013-03-11 08:05:02,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [7cc89fa6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 22c7f47f >2013-03-11 08:05:02,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [7cc89fa6] FINISH, GetHardwareInfoVDSCommand, log id: 22c7f47f >2013-03-11 08:05:02,950 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-96) [1a3f6e66] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 08:05:02,953 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-96) [2d10dd69] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 08:05:02,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 124eb6e7 >2013-03-11 08:05:08,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 08:05:08,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:05:08,975 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 08:05:08,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 124eb6e7 >2013-03-11 08:05:08,979 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [1452f5cd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:05:08,992 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [1452f5cd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4f76f48b >2013-03-11 08:05:08,992 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [1452f5cd] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 08:05:08,992 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [1452f5cd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4f76f48b >2013-03-11 08:05:08,994 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [1452f5cd] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 08:05:08,994 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [2d10dd69] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 08:05:09,004 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-96) [76ee5a62] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:05:09,005 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [76ee5a62] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6a24184c >2013-03-11 08:05:09,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [76ee5a62] FINISH, SetVdsStatusVDSCommand, log id: 6a24184c >2013-03-11 08:05:09,013 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-96) [76ee5a62] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 08:05:09,036 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-96) [2a0eeec9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:05:09,038 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-96) [47ada7d0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:05:09,039 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-96) [47ada7d0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 08:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-31) [65d12966] Autorecovering 1 hosts >2013-03-11 08:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-31) [65d12966] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-31) [7cf8300] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-31) [7cf8300] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [7cf8300] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 504d7d79 >2013-03-11 08:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [7cf8300] FINISH, SetVdsStatusVDSCommand, log id: 504d7d79 >2013-03-11 08:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-31) [7cf8300] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3ae11a05 >2013-03-11 08:10:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [7cf8300] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7047a3e6 >2013-03-11 08:10:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [7cf8300] FINISH, GetHardwareInfoVDSCommand, log id: 7047a3e6 >2013-03-11 08:10:00,258 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-31) [211e57d8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:10:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-31) [25ccbfc9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:10:00,262 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-31) [25ccbfc9] FINISH, ActivateVdsVDSCommand, log id: 3ae11a05 >2013-03-11 08:10:00,282 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-31) [25ccbfc9] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:10:00,285 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-31) [25ccbfc9] Autorecovering 1 storage domains >2013-03-11 08:10:00,285 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-31) [25ccbfc9] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 08:10:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-31) [12ddbded] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:10:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-31) [12ddbded] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 8:10 AM >2013-03-11 08:10:00,289 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-31) [12ddbded] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 8:10 AM >2013-03-11 08:10:00,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [23334eb1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1e12868b >2013-03-11 08:10:00,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [23334eb1] FINISH, GetHardwareInfoVDSCommand, log id: 1e12868b >2013-03-11 08:10:00,359 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-76) [1ddcf583] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 08:10:00,363 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-76) [447646d2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 08:10:00,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 21e81ca0 >2013-03-11 08:10:08,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 08:10:08,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:10:08,369 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 08:10:08,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 21e81ca0 >2013-03-11 08:10:08,372 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [5325f256] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:10:08,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5325f256] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7a5eca10 >2013-03-11 08:10:08,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [5325f256] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 08:10:08,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5325f256] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7a5eca10 >2013-03-11 08:10:08,388 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [5325f256] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 08:10:08,389 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [447646d2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 08:10:08,399 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-76) [d5548dd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:10:08,400 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [d5548dd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 38cffd01 >2013-03-11 08:10:08,401 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [d5548dd] FINISH, SetVdsStatusVDSCommand, log id: 38cffd01 >2013-03-11 08:10:08,407 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-76) [d5548dd] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 08:10:08,429 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-76) [113fed8c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:10:08,432 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-76) [6b031400] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:10:08,433 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-76) [6b031400] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 08:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [36b64128] Autorecovering 1 hosts >2013-03-11 08:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [36b64128] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [42d3ae8b] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [42d3ae8b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [42d3ae8b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: a3df4ac >2013-03-11 08:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [42d3ae8b] FINISH, SetVdsStatusVDSCommand, log id: a3df4ac >2013-03-11 08:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-44) [42d3ae8b] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1de8a96c >2013-03-11 08:15:00,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [42d3ae8b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6352c716 >2013-03-11 08:15:00,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [42d3ae8b] FINISH, GetHardwareInfoVDSCommand, log id: 6352c716 >2013-03-11 08:15:00,187 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-44) [7f82d1ed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:15:00,189 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-44) [5db60c32] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:15:00,191 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-44) [5db60c32] FINISH, ActivateVdsVDSCommand, log id: 1de8a96c >2013-03-11 08:15:00,210 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [5db60c32] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:15:00,211 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-44) [5db60c32] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 08:15:00,213 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [5db60c32] Autorecovering 1 storage domains >2013-03-11 08:15:00,213 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [5db60c32] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 08:15:00,214 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [43f1b61c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:15:00,214 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [43f1b61c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 8:15 AM >2013-03-11 08:15:00,217 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [43f1b61c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 8:15 AM >2013-03-11 08:15:02,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [6206e7e8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 41fb0bfe >2013-03-11 08:15:02,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [6206e7e8] FINISH, GetHardwareInfoVDSCommand, log id: 41fb0bfe >2013-03-11 08:15:02,718 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-70) [57592956] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 08:15:02,721 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-70) [16e38370] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 08:15:02,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 49946a68 >2013-03-11 08:15:09,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 08:15:09,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:15:09,711 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 08:15:09,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 49946a68 >2013-03-11 08:15:09,715 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [5262179e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:15:09,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5262179e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1fc39095 >2013-03-11 08:15:09,729 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [5262179e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 08:15:09,729 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5262179e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1fc39095 >2013-03-11 08:15:09,731 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [5262179e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 08:15:09,731 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [16e38370] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 08:15:09,740 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-70) [2a7166ee] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:15:09,742 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [2a7166ee] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2547311f >2013-03-11 08:15:09,743 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [2a7166ee] FINISH, SetVdsStatusVDSCommand, log id: 2547311f >2013-03-11 08:15:09,748 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-70) [2a7166ee] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 08:15:09,771 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-70) [23428317] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:15:09,774 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-70) [22b4a4bf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:15:09,775 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-70) [22b4a4bf] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 08:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [322cd7cc] Autorecovering 1 hosts >2013-03-11 08:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [322cd7cc] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [42ab5c01] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [42ab5c01] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [42ab5c01] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4590266d >2013-03-11 08:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [42ab5c01] FINISH, SetVdsStatusVDSCommand, log id: 4590266d >2013-03-11 08:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [42ab5c01] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 345e0e40 >2013-03-11 08:20:00,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [42ab5c01] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 48628cb >2013-03-11 08:20:00,598 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [42ab5c01] FINISH, GetHardwareInfoVDSCommand, log id: 48628cb >2013-03-11 08:20:00,644 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-79) [45f04e4f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:20:00,646 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-79) [2b8f16d9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:20:00,648 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [2b8f16d9] FINISH, ActivateVdsVDSCommand, log id: 345e0e40 >2013-03-11 08:20:00,667 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [2b8f16d9] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:20:00,668 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-79) [2b8f16d9] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 08:20:00,669 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [2b8f16d9] Autorecovering 1 storage domains >2013-03-11 08:20:00,669 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [2b8f16d9] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 08:20:00,670 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [7afa160f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:20:00,670 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [7afa160f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 8:20 AM >2013-03-11 08:20:00,673 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [7afa160f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 8:20 AM >2013-03-11 08:20:01,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [43cbe870] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 28b9032d >2013-03-11 08:20:01,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [43cbe870] FINISH, GetHardwareInfoVDSCommand, log id: 28b9032d >2013-03-11 08:20:01,060 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-82) [5d02c66d] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 08:20:01,063 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-82) [1aaeae29] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 08:20:01,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1046fc8e >2013-03-11 08:20:07,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 08:20:07,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:20:07,049 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 08:20:07,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 1046fc8e >2013-03-11 08:20:07,053 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [3b97a593] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:20:07,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [3b97a593] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 39551bd >2013-03-11 08:20:07,066 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [3b97a593] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 08:20:07,067 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [3b97a593] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 39551bd >2013-03-11 08:20:07,069 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [3b97a593] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 08:20:07,069 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [1aaeae29] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 08:20:07,078 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-82) [382de5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:20:07,080 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [382de5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 52abc7a2 >2013-03-11 08:20:07,081 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [382de5] FINISH, SetVdsStatusVDSCommand, log id: 52abc7a2 >2013-03-11 08:20:07,087 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-82) [382de5] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 08:20:07,110 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-82) [537ea87f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:20:07,113 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-82) [35412d54] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:20:07,114 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-82) [35412d54] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 08:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [32404736] Autorecovering 1 hosts >2013-03-11 08:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [32404736] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-7) [6903c884] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-7) [6903c884] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:25:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [6903c884] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 10a15520 >2013-03-11 08:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [6903c884] FINISH, SetVdsStatusVDSCommand, log id: 10a15520 >2013-03-11 08:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-7) [6903c884] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 597797ff >2013-03-11 08:25:00,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [6903c884] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2d027d1c >2013-03-11 08:25:00,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [6903c884] FINISH, GetHardwareInfoVDSCommand, log id: 2d027d1c >2013-03-11 08:25:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-7) [4dabc8a7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:25:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-7) [7e9ff902] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:25:00,095 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-7) [7e9ff902] FINISH, ActivateVdsVDSCommand, log id: 597797ff >2013-03-11 08:25:00,114 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-7) [7e9ff902] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:25:00,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-7) [7e9ff902] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 08:25:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [7e9ff902] Autorecovering 1 storage domains >2013-03-11 08:25:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [7e9ff902] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 08:25:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-7) [1dd97d46] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:25:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-7) [1dd97d46] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 8:25 AM >2013-03-11 08:25:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-7) [1dd97d46] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 8:25 AM >2013-03-11 08:25:01,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [1a7544b0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5ce6a2ca >2013-03-11 08:25:01,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [1a7544b0] FINISH, GetHardwareInfoVDSCommand, log id: 5ce6a2ca >2013-03-11 08:25:01,392 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-78) [5b6f7620] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 08:25:01,396 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-78) [44dcb94e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 08:25:01,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 25b07e8a >2013-03-11 08:25:07,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 08:25:07,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:25:07,415 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 08:25:07,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 25b07e8a >2013-03-11 08:25:07,419 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [50701739] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:25:07,432 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [50701739] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 27a6eab7 >2013-03-11 08:25:07,433 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [50701739] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 08:25:07,433 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [50701739] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 27a6eab7 >2013-03-11 08:25:07,435 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [50701739] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 08:25:07,435 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [44dcb94e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 08:25:07,445 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-78) [4a445b10] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:25:07,446 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [4a445b10] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 56d25a2 >2013-03-11 08:25:07,447 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [4a445b10] FINISH, SetVdsStatusVDSCommand, log id: 56d25a2 >2013-03-11 08:25:07,453 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-78) [4a445b10] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 08:25:07,477 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-78) [7fb02ea9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:25:07,479 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-78) [1fdb2f9b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:25:07,481 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-78) [1fdb2f9b] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 08:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [347a0f47] Autorecovering 1 hosts >2013-03-11 08:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [347a0f47] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-48) [31ebf3c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-48) [31ebf3c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [31ebf3c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6d16d890 >2013-03-11 08:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [31ebf3c] FINISH, SetVdsStatusVDSCommand, log id: 6d16d890 >2013-03-11 08:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-48) [31ebf3c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7ef5dafe >2013-03-11 08:30:00,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [31ebf3c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 597d980b >2013-03-11 08:30:00,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [31ebf3c] FINISH, GetHardwareInfoVDSCommand, log id: 597d980b >2013-03-11 08:30:00,097 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-48) [54c86ffa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:30:00,099 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-48) [59c09c93] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:30:00,101 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-48) [59c09c93] FINISH, ActivateVdsVDSCommand, log id: 7ef5dafe >2013-03-11 08:30:00,120 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-48) [59c09c93] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:30:00,121 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-48) [59c09c93] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 08:30:00,122 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [59c09c93] Autorecovering 1 storage domains >2013-03-11 08:30:00,122 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-48) [59c09c93] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 08:30:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-48) [640da200] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:30:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-48) [640da200] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 8:30 AM >2013-03-11 08:30:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-48) [640da200] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 8:30 AM >2013-03-11 08:30:01,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [319b33a5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1eb3f3e1 >2013-03-11 08:30:01,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [319b33a5] FINISH, GetHardwareInfoVDSCommand, log id: 1eb3f3e1 >2013-03-11 08:30:01,761 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-86) [392b6b7] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 08:30:01,764 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-86) [59524eea] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 08:30:01,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3194d2b1 >2013-03-11 08:30:07,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 08:30:07,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:30:07,795 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 08:30:07,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 3194d2b1 >2013-03-11 08:30:07,799 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [1f7d4856] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:30:07,812 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [1f7d4856] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 50972aa7 >2013-03-11 08:30:07,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [1f7d4856] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 08:30:07,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [1f7d4856] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 50972aa7 >2013-03-11 08:30:07,815 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [1f7d4856] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 08:30:07,815 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [59524eea] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 08:30:07,824 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-86) [5b820a8c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:30:07,826 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [5b820a8c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6aae495c >2013-03-11 08:30:07,827 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [5b820a8c] FINISH, SetVdsStatusVDSCommand, log id: 6aae495c >2013-03-11 08:30:07,833 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-86) [5b820a8c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 08:30:07,857 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-86) [ef24748] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:30:07,859 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-86) [271504c2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:30:07,861 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-86) [271504c2] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 08:30:33,877 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-94) [452f63bc] Start refreshing all users data >2013-03-11 08:30:35,067 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [137d1c0a] Attempting to update VMs/Templates Ovf. >2013-03-11 08:30:35,068 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [137d1c0a] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 08:30:35,069 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [137d1c0a] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 08:30:35,069 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [137d1c0a] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 08:30:35,069 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [137d1c0a] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 08:30:35,070 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [137d1c0a] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 08:30:35,070 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-24) [137d1c0a] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 08:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [7afa160f] Autorecovering 1 hosts >2013-03-11 08:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [7afa160f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [648a103e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [648a103e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [648a103e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 65305fab >2013-03-11 08:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [648a103e] FINISH, SetVdsStatusVDSCommand, log id: 65305fab >2013-03-11 08:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [648a103e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 780d2ad >2013-03-11 08:35:00,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [648a103e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5fe037b2 >2013-03-11 08:35:00,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [648a103e] FINISH, GetHardwareInfoVDSCommand, log id: 5fe037b2 >2013-03-11 08:35:00,190 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-79) [79ba6a5d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:35:00,192 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-79) [13813798] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:35:00,193 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [13813798] FINISH, ActivateVdsVDSCommand, log id: 780d2ad >2013-03-11 08:35:00,212 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [13813798] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:35:00,213 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-79) [13813798] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 08:35:00,215 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [13813798] Autorecovering 1 storage domains >2013-03-11 08:35:00,215 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [13813798] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 08:35:00,215 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [50f7ef2f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:35:00,216 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [50f7ef2f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 8:35 AM >2013-03-11 08:35:00,218 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [50f7ef2f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 8:35 AM >2013-03-11 08:35:02,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [762f7c4f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 27228f44 >2013-03-11 08:35:02,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [762f7c4f] FINISH, GetHardwareInfoVDSCommand, log id: 27228f44 >2013-03-11 08:35:02,142 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-66) [6e408351] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 08:35:02,146 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-66) [3be80079] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 08:35:02,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4542dfb3 >2013-03-11 08:35:09,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 08:35:09,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:35:09,143 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 08:35:09,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 4542dfb3 >2013-03-11 08:35:09,146 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [134882eb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:35:09,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [134882eb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 71e84b0b >2013-03-11 08:35:09,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [134882eb] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 08:35:09,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [134882eb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 71e84b0b >2013-03-11 08:35:09,162 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [134882eb] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 08:35:09,162 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [3be80079] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 08:35:09,171 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-66) [34b1638e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:35:09,173 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [34b1638e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5531b66d >2013-03-11 08:35:09,174 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [34b1638e] FINISH, SetVdsStatusVDSCommand, log id: 5531b66d >2013-03-11 08:35:09,179 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-66) [34b1638e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 08:35:09,203 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-66) [5e6b6654] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:35:09,205 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-66) [620d05a4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:35:09,206 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-66) [620d05a4] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 08:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [1dd97d46] Autorecovering 1 hosts >2013-03-11 08:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [1dd97d46] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-7) [4bcdd011] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-7) [4bcdd011] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [4bcdd011] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 451e8226 >2013-03-11 08:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [4bcdd011] FINISH, SetVdsStatusVDSCommand, log id: 451e8226 >2013-03-11 08:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-7) [4bcdd011] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 8770b60 >2013-03-11 08:40:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [4bcdd011] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3a125b9 >2013-03-11 08:40:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [4bcdd011] FINISH, GetHardwareInfoVDSCommand, log id: 3a125b9 >2013-03-11 08:40:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-7) [56c215f5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:40:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-7) [24b169c0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:40:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-7) [24b169c0] FINISH, ActivateVdsVDSCommand, log id: 8770b60 >2013-03-11 08:40:00,219 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-7) [24b169c0] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:40:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-7) [24b169c0] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 08:40:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [24b169c0] Autorecovering 1 storage domains >2013-03-11 08:40:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [24b169c0] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 08:40:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-7) [32fba2a4] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:40:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-7) [32fba2a4] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 8:40 AM >2013-03-11 08:40:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-7) [32fba2a4] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 8:40 AM >2013-03-11 08:40:00,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [15953ece] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7c63a0e2 >2013-03-11 08:40:00,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [15953ece] FINISH, GetHardwareInfoVDSCommand, log id: 7c63a0e2 >2013-03-11 08:40:00,487 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-34) [622dacef] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 08:40:00,491 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-34) [802a6e0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 08:40:00,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4866f6f8 >2013-03-11 08:40:07,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 08:40:07,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:40:07,501 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 08:40:07,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 4866f6f8 >2013-03-11 08:40:07,505 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [6772a0c2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:40:07,518 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [6772a0c2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 340de7c >2013-03-11 08:40:07,518 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [6772a0c2] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 08:40:07,518 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [6772a0c2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 340de7c >2013-03-11 08:40:07,520 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [6772a0c2] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 08:40:07,521 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [802a6e0] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 08:40:07,534 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-34) [75cf8424] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:40:07,536 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [75cf8424] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 13838899 >2013-03-11 08:40:07,537 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [75cf8424] FINISH, SetVdsStatusVDSCommand, log id: 13838899 >2013-03-11 08:40:07,543 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-34) [75cf8424] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 08:40:07,573 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-34) [49fc3768] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:40:07,575 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-34) [725358bc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:40:07,577 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) [725358bc] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 08:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) [6d119077] Autorecovering 1 hosts >2013-03-11 08:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) [6d119077] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:45:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-6) [3407079f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-6) [3407079f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:45:00,006 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-6) [3407079f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 34fbd344 >2013-03-11 08:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-6) [3407079f] FINISH, SetVdsStatusVDSCommand, log id: 34fbd344 >2013-03-11 08:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-6) [3407079f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 61bff6a3 >2013-03-11 08:45:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-6) [3407079f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 68068848 >2013-03-11 08:45:00,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-6) [3407079f] FINISH, GetHardwareInfoVDSCommand, log id: 68068848 >2013-03-11 08:45:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-6) [35348202] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:45:00,206 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-6) [474745ad] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:45:00,208 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-6) [474745ad] FINISH, ActivateVdsVDSCommand, log id: 61bff6a3 >2013-03-11 08:45:00,227 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-6) [474745ad] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:45:00,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-6) [474745ad] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 08:45:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) [474745ad] Autorecovering 1 storage domains >2013-03-11 08:45:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) [474745ad] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 08:45:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-6) [645e4926] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:45:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-6) [645e4926] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 8:45 AM >2013-03-11 08:45:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-6) [645e4926] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 8:45 AM >2013-03-11 08:45:01,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [7be57232] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5deacdb7 >2013-03-11 08:45:01,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [7be57232] FINISH, GetHardwareInfoVDSCommand, log id: 5deacdb7 >2013-03-11 08:45:01,855 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-84) [54d0198d] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 08:45:01,858 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-84) [78e48627] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 08:45:01,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6fb7f94c >2013-03-11 08:45:08,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 08:45:08,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:45:08,856 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 08:45:08,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 6fb7f94c >2013-03-11 08:45:08,860 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [48cd77b0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:45:08,873 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [48cd77b0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 499872f1 >2013-03-11 08:45:08,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [48cd77b0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 08:45:08,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [48cd77b0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 499872f1 >2013-03-11 08:45:08,878 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [78e48627] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 08:45:08,888 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-84) [6d65c68b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:45:08,890 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [6d65c68b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7ffcfccf >2013-03-11 08:45:08,891 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [6d65c68b] FINISH, SetVdsStatusVDSCommand, log id: 7ffcfccf >2013-03-11 08:45:08,897 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-84) [6d65c68b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 08:45:08,928 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-84) [70cf7142] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:45:08,931 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-84) [e79098a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:45:08,932 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-84) [e79098a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 08:50:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) [7d18a3ff] Autorecovering 1 hosts >2013-03-11 08:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) [7d18a3ff] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-11) [2c0c1f6e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-11) [2c0c1f6e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:50:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [2c0c1f6e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 63c4a470 >2013-03-11 08:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [2c0c1f6e] FINISH, SetVdsStatusVDSCommand, log id: 63c4a470 >2013-03-11 08:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-11) [2c0c1f6e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1d668d28 >2013-03-11 08:50:00,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [2c0c1f6e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 45f61f5d >2013-03-11 08:50:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [2c0c1f6e] FINISH, GetHardwareInfoVDSCommand, log id: 45f61f5d >2013-03-11 08:50:00,182 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-11) [61477fa4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:50:00,185 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-11) [2d1c05d1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:50:00,186 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-11) [2d1c05d1] FINISH, ActivateVdsVDSCommand, log id: 1d668d28 >2013-03-11 08:50:00,207 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-11) [2d1c05d1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:50:00,208 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-11) [2d1c05d1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 08:50:00,209 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) [2d1c05d1] Autorecovering 1 storage domains >2013-03-11 08:50:00,209 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-11) [2d1c05d1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 08:50:00,210 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-11) [27e50313] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:50:00,210 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-11) [27e50313] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 8:50 AM >2013-03-11 08:50:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-11) [27e50313] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 8:50 AM >2013-03-11 08:50:00,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [5c84216c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3b3e0668 >2013-03-11 08:50:00,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [5c84216c] FINISH, GetHardwareInfoVDSCommand, log id: 3b3e0668 >2013-03-11 08:50:00,287 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-90) [668aeade] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 08:50:00,290 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-90) [2ba4668f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 08:50:00,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5808e05c >2013-03-11 08:50:07,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 08:50:07,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:50:07,297 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 08:50:07,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 5808e05c >2013-03-11 08:50:07,301 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [61d3b725] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:50:07,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [61d3b725] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4362d7c >2013-03-11 08:50:07,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [61d3b725] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 08:50:07,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [61d3b725] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4362d7c >2013-03-11 08:50:07,317 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [61d3b725] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 08:50:07,318 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [2ba4668f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 08:50:07,327 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-90) [31535dc9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:50:07,329 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [31535dc9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 43f3135 >2013-03-11 08:50:07,330 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [31535dc9] FINISH, SetVdsStatusVDSCommand, log id: 43f3135 >2013-03-11 08:50:07,336 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [31535dc9] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 08:50:07,360 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-90) [5629f9f4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:50:07,363 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-90) [1c5fdbf2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:50:07,364 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-90) [1c5fdbf2] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 08:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [db67848] Autorecovering 1 hosts >2013-03-11 08:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [db67848] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [52891882] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [52891882] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [52891882] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3859c7bb >2013-03-11 08:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [52891882] FINISH, SetVdsStatusVDSCommand, log id: 3859c7bb >2013-03-11 08:55:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-47) [52891882] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 10b55dca >2013-03-11 08:55:00,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [52891882] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: df3544b >2013-03-11 08:55:00,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [52891882] FINISH, GetHardwareInfoVDSCommand, log id: df3544b >2013-03-11 08:55:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [3e38ac70] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:55:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [75694ef5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:55:00,205 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-47) [75694ef5] FINISH, ActivateVdsVDSCommand, log id: 10b55dca >2013-03-11 08:55:00,225 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-47) [75694ef5] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 08:55:00,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [75694ef5] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 08:55:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [75694ef5] Autorecovering 1 storage domains >2013-03-11 08:55:00,227 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-47) [75694ef5] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 08:55:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [2a02aafc] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:55:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [2a02aafc] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 8:55 AM >2013-03-11 08:55:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-47) [2a02aafc] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 8:55 AM >2013-03-11 08:55:01,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [32fba2a4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 342ff59 >2013-03-11 08:55:01,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [32fba2a4] FINISH, GetHardwareInfoVDSCommand, log id: 342ff59 >2013-03-11 08:55:01,639 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-7) [61412a8] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 08:55:01,643 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-7) [2d134053] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 08:55:01,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: d7e4d7 >2013-03-11 08:55:06,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 08:55:06,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 08:55:06,655 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 08:55:06,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: d7e4d7 >2013-03-11 08:55:06,659 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [15ca7b5a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 08:55:06,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [15ca7b5a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2f2c201c >2013-03-11 08:55:06,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [15ca7b5a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 08:55:06,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [15ca7b5a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2f2c201c >2013-03-11 08:55:06,676 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [15ca7b5a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 08:55:06,677 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [2d134053] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 08:55:06,689 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-7) [338ba59d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:55:06,691 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [338ba59d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2b94c4f0 >2013-03-11 08:55:06,692 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [338ba59d] FINISH, SetVdsStatusVDSCommand, log id: 2b94c4f0 >2013-03-11 08:55:06,698 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-7) [338ba59d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 08:55:06,733 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-7) [1aa7bfae] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:55:06,735 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-7) [52915c3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 08:55:06,737 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) [52915c3] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 09:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [1c369396] Autorecovering 1 hosts >2013-03-11 09:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [1c369396] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [52047d18] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [52047d18] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [52047d18] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6dadadd5 >2013-03-11 09:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [52047d18] FINISH, SetVdsStatusVDSCommand, log id: 6dadadd5 >2013-03-11 09:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-19) [52047d18] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 682db996 >2013-03-11 09:00:00,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [52047d18] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 440b7913 >2013-03-11 09:00:00,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [52047d18] FINISH, GetHardwareInfoVDSCommand, log id: 440b7913 >2013-03-11 09:00:00,209 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-19) [1b53649a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:00:00,211 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-19) [6a8972ba] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:00:00,213 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-19) [6a8972ba] FINISH, ActivateVdsVDSCommand, log id: 682db996 >2013-03-11 09:00:00,232 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [6a8972ba] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:00:00,233 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-19) [6a8972ba] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 09:00:00,234 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [6a8972ba] Autorecovering 1 storage domains >2013-03-11 09:00:00,234 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [6a8972ba] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 09:00:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [50a62fe] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:00:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [50a62fe] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 9:00 AM >2013-03-11 09:00:00,237 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [50a62fe] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 9:00 AM >2013-03-11 09:00:00,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [3a30c61c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3bf4b871 >2013-03-11 09:00:00,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [3a30c61c] FINISH, GetHardwareInfoVDSCommand, log id: 3bf4b871 >2013-03-11 09:00:01,015 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-99) [7cea7300] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 09:00:01,019 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-99) [13dbaad5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 09:00:01,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 28d2649 >2013-03-11 09:00:07,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 09:00:07,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:00:07,055 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 09:00:07,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 28d2649 >2013-03-11 09:00:07,058 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [4ef34f87] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:00:07,072 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4ef34f87] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c6f82ae >2013-03-11 09:00:07,072 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [4ef34f87] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 09:00:07,072 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4ef34f87] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c6f82ae >2013-03-11 09:00:07,074 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [4ef34f87] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 09:00:07,075 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [13dbaad5] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 09:00:07,084 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-99) [ee03c71] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:00:07,086 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [ee03c71] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 57e8f363 >2013-03-11 09:00:07,087 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [ee03c71] FINISH, SetVdsStatusVDSCommand, log id: 57e8f363 >2013-03-11 09:00:07,093 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-99) [ee03c71] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 09:00:07,118 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-99) [3d45a8b3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:00:07,120 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-99) [65bf0f9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:00:07,121 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-99) [65bf0f9] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 09:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [7e4351f5] Autorecovering 1 hosts >2013-03-11 09:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [7e4351f5] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [377bfff] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [377bfff] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [377bfff] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 467c20fb >2013-03-11 09:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [377bfff] FINISH, SetVdsStatusVDSCommand, log id: 467c20fb >2013-03-11 09:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-32) [377bfff] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 76d80d1d >2013-03-11 09:05:00,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [377bfff] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3d73266b >2013-03-11 09:05:00,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [377bfff] FINISH, GetHardwareInfoVDSCommand, log id: 3d73266b >2013-03-11 09:05:00,192 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-32) [6c9d0414] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:05:00,194 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-32) [14f8c00f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:05:00,196 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-32) [14f8c00f] FINISH, ActivateVdsVDSCommand, log id: 76d80d1d >2013-03-11 09:05:00,215 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [14f8c00f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:05:00,216 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-32) [14f8c00f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 09:05:00,217 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [14f8c00f] Autorecovering 1 storage domains >2013-03-11 09:05:00,217 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [14f8c00f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 09:05:00,218 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [668b4021] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:05:00,218 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [668b4021] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 9:05 AM >2013-03-11 09:05:00,221 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [668b4021] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 9:05 AM >2013-03-11 09:05:01,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [27e50313] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 738b57ab >2013-03-11 09:05:01,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [27e50313] FINISH, GetHardwareInfoVDSCommand, log id: 738b57ab >2013-03-11 09:05:01,402 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-11) [5940c736] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 09:05:01,406 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-11) [41c555ec] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 09:05:01,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1821d7eb >2013-03-11 09:05:08,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 09:05:08,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:05:08,426 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 09:05:08,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 1821d7eb >2013-03-11 09:05:08,429 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [229b4c2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:05:08,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [229b4c2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: a862f74 >2013-03-11 09:05:08,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [229b4c2] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 09:05:08,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [229b4c2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: a862f74 >2013-03-11 09:05:08,445 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [229b4c2] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 09:05:08,446 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [41c555ec] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 09:05:08,456 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-11) [20e03176] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:05:08,457 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [20e03176] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 18c0fb1e >2013-03-11 09:05:08,458 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [20e03176] FINISH, SetVdsStatusVDSCommand, log id: 18c0fb1e >2013-03-11 09:05:08,464 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-11) [20e03176] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 09:05:08,488 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-11) [4b9be4f4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:05:08,491 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-11) [51ddbdd7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:05:08,492 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-11) [51ddbdd7] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 09:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [27d2a58d] Autorecovering 1 hosts >2013-03-11 09:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [27d2a58d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [46acf0e0] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [46acf0e0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:10:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [46acf0e0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3029d769 >2013-03-11 09:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [46acf0e0] FINISH, SetVdsStatusVDSCommand, log id: 3029d769 >2013-03-11 09:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-64) [46acf0e0] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6c7a6baf >2013-03-11 09:10:00,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [46acf0e0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 76228f40 >2013-03-11 09:10:00,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [46acf0e0] FINISH, GetHardwareInfoVDSCommand, log id: 76228f40 >2013-03-11 09:10:00,619 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-64) [1adea8d4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:10:00,621 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-64) [605ff60] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:10:00,623 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-64) [605ff60] FINISH, ActivateVdsVDSCommand, log id: 6c7a6baf >2013-03-11 09:10:00,642 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-64) [605ff60] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:10:00,643 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-64) [605ff60] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 09:10:00,644 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [605ff60] Autorecovering 1 storage domains >2013-03-11 09:10:00,644 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-64) [605ff60] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 09:10:00,645 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [1f24fadb] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:10:00,645 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [1f24fadb] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 9:10 AM >2013-03-11 09:10:00,648 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-64) [1f24fadb] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 9:10 AM >2013-03-11 09:10:02,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [2a02aafc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3ce876f8 >2013-03-11 09:10:02,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [2a02aafc] FINISH, GetHardwareInfoVDSCommand, log id: 3ce876f8 >2013-03-11 09:10:02,772 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-47) [23d3ea2c] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 09:10:02,776 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-47) [46310b64] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 09:10:02,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4e3fc293 >2013-03-11 09:10:08,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 09:10:08,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:10:08,779 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 09:10:08,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 4e3fc293 >2013-03-11 09:10:08,783 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [356dc1bd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:10:08,796 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [356dc1bd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 46272d94 >2013-03-11 09:10:08,796 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [356dc1bd] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 09:10:08,796 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [356dc1bd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 46272d94 >2013-03-11 09:10:08,798 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [356dc1bd] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 09:10:08,799 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [46310b64] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 09:10:08,808 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-47) [40556af2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:10:08,809 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [40556af2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 104588e5 >2013-03-11 09:10:08,811 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [40556af2] FINISH, SetVdsStatusVDSCommand, log id: 104588e5 >2013-03-11 09:10:08,817 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [40556af2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 09:10:08,841 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [6a1beaab] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:10:08,843 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [514c8dcb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:10:08,844 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-47) [514c8dcb] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 09:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [20966fe5] Autorecovering 1 hosts >2013-03-11 09:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [20966fe5] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [272280cf] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [272280cf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [272280cf] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5124c9bc >2013-03-11 09:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [272280cf] FINISH, SetVdsStatusVDSCommand, log id: 5124c9bc >2013-03-11 09:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [272280cf] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5312b0b2 >2013-03-11 09:15:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [272280cf] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4903bf >2013-03-11 09:15:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [272280cf] FINISH, GetHardwareInfoVDSCommand, log id: 4903bf >2013-03-11 09:15:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [193d4ca7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:15:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [4f91b083] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:15:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [4f91b083] FINISH, ActivateVdsVDSCommand, log id: 5312b0b2 >2013-03-11 09:15:00,111 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [4f91b083] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:15:00,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [4f91b083] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 09:15:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [4f91b083] Autorecovering 1 storage domains >2013-03-11 09:15:00,113 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [4f91b083] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 09:15:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [4f44faa2] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:15:00,114 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [4f44faa2] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 9:15 AM >2013-03-11 09:15:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [4f44faa2] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 9:15 AM >2013-03-11 09:15:00,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [66d774e0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1f5424cd >2013-03-11 09:15:00,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [66d774e0] FINISH, GetHardwareInfoVDSCommand, log id: 1f5424cd >2013-03-11 09:15:00,191 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-91) [e6e3848] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 09:15:00,194 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-91) [1df8e2c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 09:15:00,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1b15f13b >2013-03-11 09:15:06,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 09:15:06,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:15:06,175 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 09:15:06,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 1b15f13b >2013-03-11 09:15:06,179 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [66675253] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:15:06,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [66675253] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4ae0e93f >2013-03-11 09:15:06,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [66675253] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 09:15:06,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [66675253] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4ae0e93f >2013-03-11 09:15:06,194 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [66675253] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 09:15:06,195 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [1df8e2c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 09:15:06,204 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-91) [29a001c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:15:06,205 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [29a001c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5166829c >2013-03-11 09:15:06,207 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [29a001c] FINISH, SetVdsStatusVDSCommand, log id: 5166829c >2013-03-11 09:15:06,212 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-91) [29a001c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 09:15:06,235 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-91) [737584cf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:15:06,237 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-91) [90bee01] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:15:06,239 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-91) [90bee01] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 09:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [e6ba8a7] Autorecovering 1 hosts >2013-03-11 09:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [e6ba8a7] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-53) [3b9789ed] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-53) [3b9789ed] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [3b9789ed] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 11d271ef >2013-03-11 09:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [3b9789ed] FINISH, SetVdsStatusVDSCommand, log id: 11d271ef >2013-03-11 09:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-53) [3b9789ed] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7aabaca8 >2013-03-11 09:20:00,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [3b9789ed] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5fe5def1 >2013-03-11 09:20:00,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [3b9789ed] FINISH, GetHardwareInfoVDSCommand, log id: 5fe5def1 >2013-03-11 09:20:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-53) [37847cb3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:20:00,262 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-53) [2e486614] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:20:00,264 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-53) [2e486614] FINISH, ActivateVdsVDSCommand, log id: 7aabaca8 >2013-03-11 09:20:00,283 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-53) [2e486614] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:20:00,284 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-53) [2e486614] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 09:20:00,285 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [2e486614] Autorecovering 1 storage domains >2013-03-11 09:20:00,285 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [2e486614] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 09:20:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-53) [54c6659e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:20:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-53) [54c6659e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 9:20 AM >2013-03-11 09:20:00,289 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-53) [54c6659e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 9:20 AM >2013-03-11 09:20:00,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [51e25f53] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 68224980 >2013-03-11 09:20:00,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [51e25f53] FINISH, GetHardwareInfoVDSCommand, log id: 68224980 >2013-03-11 09:20:00,521 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-57) [45b68f7a] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 09:20:00,524 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-57) [3ec6f3b3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 09:20:00,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 431b65c9 >2013-03-11 09:20:06,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 09:20:06,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:20:06,506 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 09:20:06,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 431b65c9 >2013-03-11 09:20:06,510 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [5a570fac] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:20:06,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [5a570fac] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2288871 >2013-03-11 09:20:06,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [5a570fac] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 09:20:06,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [5a570fac] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2288871 >2013-03-11 09:20:06,527 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [5a570fac] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 09:20:06,527 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [3ec6f3b3] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 09:20:06,537 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-57) [252fcb9b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:20:06,538 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [252fcb9b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 64e6b51a >2013-03-11 09:20:06,540 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [252fcb9b] FINISH, SetVdsStatusVDSCommand, log id: 64e6b51a >2013-03-11 09:20:06,545 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-57) [252fcb9b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 09:20:06,568 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-57) [313e8cee] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:20:06,571 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-57) [574be938] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:20:06,572 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-57) [574be938] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 09:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [2cea74d1] Autorecovering 1 hosts >2013-03-11 09:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [2cea74d1] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [409845f4] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [409845f4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:25:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [409845f4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 224b9ab5 >2013-03-11 09:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [409845f4] FINISH, SetVdsStatusVDSCommand, log id: 224b9ab5 >2013-03-11 09:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [409845f4] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 15505a13 >2013-03-11 09:25:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [409845f4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: e52724b >2013-03-11 09:25:00,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [409845f4] FINISH, GetHardwareInfoVDSCommand, log id: e52724b >2013-03-11 09:25:00,290 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [583c9bbe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:25:00,293 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [1e283464] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:25:00,294 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [1e283464] FINISH, ActivateVdsVDSCommand, log id: 15505a13 >2013-03-11 09:25:00,314 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [1e283464] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:25:00,315 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-8) [1e283464] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 09:25:00,316 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [1e283464] Autorecovering 1 storage domains >2013-03-11 09:25:00,316 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [1e283464] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 09:25:00,317 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [4f64ac06] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:25:00,317 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [4f64ac06] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 9:25 AM >2013-03-11 09:25:00,320 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [4f64ac06] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 9:25 AM >2013-03-11 09:25:00,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [68e3aa6a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1ff83c63 >2013-03-11 09:25:00,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [68e3aa6a] FINISH, GetHardwareInfoVDSCommand, log id: 1ff83c63 >2013-03-11 09:25:00,860 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-97) [7fb9512e] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 09:25:00,864 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-97) [7f0ed74b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 09:25:00,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 54d17232 >2013-03-11 09:25:06,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 09:25:06,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:25:06,850 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 09:25:06,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 54d17232 >2013-03-11 09:25:06,855 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [2283a9a0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:25:06,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [2283a9a0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7590f194 >2013-03-11 09:25:06,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [2283a9a0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 09:25:06,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [2283a9a0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7590f194 >2013-03-11 09:25:06,870 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [2283a9a0] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 09:25:06,871 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [7f0ed74b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 09:25:06,880 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-97) [2ce42768] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:25:06,881 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [2ce42768] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 54497334 >2013-03-11 09:25:06,883 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [2ce42768] FINISH, SetVdsStatusVDSCommand, log id: 54497334 >2013-03-11 09:25:06,889 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-97) [2ce42768] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 09:25:06,912 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-97) [c1b490] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:25:06,915 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-97) [1b3a9f02] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:25:06,916 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-97) [1b3a9f02] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 09:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [7ad21c25] Autorecovering 1 hosts >2013-03-11 09:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [7ad21c25] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-39) [44df3019] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-39) [44df3019] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:30:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [44df3019] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 67dc8c90 >2013-03-11 09:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [44df3019] FINISH, SetVdsStatusVDSCommand, log id: 67dc8c90 >2013-03-11 09:30:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-39) [44df3019] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 26b1e197 >2013-03-11 09:30:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [44df3019] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4f1f6ec9 >2013-03-11 09:30:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [44df3019] FINISH, GetHardwareInfoVDSCommand, log id: 4f1f6ec9 >2013-03-11 09:30:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-39) [2205d9b6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:30:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-39) [3c29177f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:30:00,199 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-39) [3c29177f] FINISH, ActivateVdsVDSCommand, log id: 26b1e197 >2013-03-11 09:30:00,218 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-39) [3c29177f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:30:00,219 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-39) [3c29177f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 09:30:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [3c29177f] Autorecovering 1 storage domains >2013-03-11 09:30:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [3c29177f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 09:30:00,221 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-39) [44f88a48] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:30:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-39) [44f88a48] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 9:30 AM >2013-03-11 09:30:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-39) [44f88a48] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 9:30 AM >2013-03-11 09:30:01,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [b903631] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5e51d7e9 >2013-03-11 09:30:01,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [b903631] FINISH, GetHardwareInfoVDSCommand, log id: 5e51d7e9 >2013-03-11 09:30:01,196 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-83) [2c9494df] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 09:30:01,199 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-83) [4133fead] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 09:30:01,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 10dd903b >2013-03-11 09:30:07,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 09:30:07,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:30:07,237 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 09:30:07,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 10dd903b >2013-03-11 09:30:07,240 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [3f0798ac] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:30:07,253 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [3f0798ac] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 71bd8c54 >2013-03-11 09:30:07,253 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [3f0798ac] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 09:30:07,253 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [3f0798ac] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 71bd8c54 >2013-03-11 09:30:07,255 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [3f0798ac] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 09:30:07,256 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [4133fead] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 09:30:07,265 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-83) [5c3b8971] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:30:07,266 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [5c3b8971] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5f0838ce >2013-03-11 09:30:07,268 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [5c3b8971] FINISH, SetVdsStatusVDSCommand, log id: 5f0838ce >2013-03-11 09:30:07,273 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-83) [5c3b8971] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 09:30:07,302 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-83) [53d27ad0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:30:07,304 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-83) [6cb94f44] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:30:07,306 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-83) [6cb94f44] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 09:30:33,879 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-85) [183a8084] Start refreshing all users data >2013-03-11 09:30:35,070 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-13) [43ef0f74] Attempting to update VMs/Templates Ovf. >2013-03-11 09:30:35,071 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-13) [43ef0f74] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 09:30:35,072 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-13) [43ef0f74] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 09:30:35,072 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-13) [43ef0f74] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 09:30:35,073 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-13) [43ef0f74] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 09:30:35,073 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-13) [43ef0f74] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 09:30:35,074 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-13) [43ef0f74] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 09:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [54c6659e] Autorecovering 1 hosts >2013-03-11 09:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [54c6659e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-53) [11ab9054] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-53) [11ab9054] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [11ab9054] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: f6a30f3 >2013-03-11 09:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [11ab9054] FINISH, SetVdsStatusVDSCommand, log id: f6a30f3 >2013-03-11 09:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-53) [11ab9054] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6ab914be >2013-03-11 09:35:00,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [11ab9054] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5441aa15 >2013-03-11 09:35:00,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [11ab9054] FINISH, GetHardwareInfoVDSCommand, log id: 5441aa15 >2013-03-11 09:35:00,296 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-53) [7c8a67a9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:35:00,299 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-53) [45f52c94] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:35:00,300 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-53) [45f52c94] FINISH, ActivateVdsVDSCommand, log id: 6ab914be >2013-03-11 09:35:00,319 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-53) [45f52c94] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:35:00,320 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-53) [45f52c94] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 09:35:00,321 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [45f52c94] Autorecovering 1 storage domains >2013-03-11 09:35:00,321 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-53) [45f52c94] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 09:35:00,322 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-53) [695ed367] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:35:00,322 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-53) [695ed367] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 9:35 AM >2013-03-11 09:35:00,325 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-53) [695ed367] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 9:35 AM >2013-03-11 09:35:01,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [4acf5364] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 705e116 >2013-03-11 09:35:01,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [4acf5364] FINISH, GetHardwareInfoVDSCommand, log id: 705e116 >2013-03-11 09:35:01,581 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-56) [7efe2a9a] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 09:35:01,584 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-56) [36b60821] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 09:35:01,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 354366ae >2013-03-11 09:35:09,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 09:35:09,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:35:09,576 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 09:35:09,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 354366ae >2013-03-11 09:35:09,579 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [6496f4fc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:35:09,593 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [6496f4fc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 14a3c757 >2013-03-11 09:35:09,593 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [6496f4fc] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 09:35:09,593 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [6496f4fc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 14a3c757 >2013-03-11 09:35:09,595 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [6496f4fc] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 09:35:09,596 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [36b60821] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 09:35:09,605 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-56) [59d9cbaf] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:35:09,606 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [59d9cbaf] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 71589b8f >2013-03-11 09:35:09,607 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [59d9cbaf] FINISH, SetVdsStatusVDSCommand, log id: 71589b8f >2013-03-11 09:35:09,613 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-56) [59d9cbaf] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 09:35:09,636 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-56) [5565196b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:35:09,639 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-56) [706addc6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:35:09,640 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-56) [706addc6] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 09:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) [37f244d0] Autorecovering 1 hosts >2013-03-11 09:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) [37f244d0] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-61) [2159326e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-61) [2159326e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [2159326e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6f83b255 >2013-03-11 09:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [2159326e] FINISH, SetVdsStatusVDSCommand, log id: 6f83b255 >2013-03-11 09:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-61) [2159326e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: f97100 >2013-03-11 09:40:00,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [2159326e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5f36d03e >2013-03-11 09:40:00,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [2159326e] FINISH, GetHardwareInfoVDSCommand, log id: 5f36d03e >2013-03-11 09:40:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-61) [6716889c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:40:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-61) [5e207961] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:40:00,205 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-61) [5e207961] FINISH, ActivateVdsVDSCommand, log id: f97100 >2013-03-11 09:40:00,224 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-61) [5e207961] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:40:00,225 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-61) [5e207961] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 09:40:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) [5e207961] Autorecovering 1 storage domains >2013-03-11 09:40:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) [5e207961] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 09:40:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-61) [79d8b398] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:40:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-61) [79d8b398] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 9:40 AM >2013-03-11 09:40:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-61) [79d8b398] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 9:40 AM >2013-03-11 09:40:00,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [4f64ac06] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 12ac8341 >2013-03-11 09:40:00,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [4f64ac06] FINISH, GetHardwareInfoVDSCommand, log id: 12ac8341 >2013-03-11 09:40:00,920 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-8) [50ece94c] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 09:40:00,924 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-8) [5c73820c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 09:40:00,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5cff7181 >2013-03-11 09:40:08,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 09:40:08,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:40:08,915 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 09:40:08,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 5cff7181 >2013-03-11 09:40:08,919 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [423860c6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:40:08,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [423860c6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1ffe0b10 >2013-03-11 09:40:08,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [423860c6] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 09:40:08,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [423860c6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1ffe0b10 >2013-03-11 09:40:08,934 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [423860c6] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 09:40:08,935 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [5c73820c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 09:40:08,945 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-8) [1097c5e9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:40:08,946 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [1097c5e9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 78f37639 >2013-03-11 09:40:08,947 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [1097c5e9] FINISH, SetVdsStatusVDSCommand, log id: 78f37639 >2013-03-11 09:40:08,953 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-8) [1097c5e9] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 09:40:08,977 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [7457baaa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:40:08,979 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [5331bd9a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:40:08,981 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-8) [5331bd9a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 09:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [ab15a99] Autorecovering 1 hosts >2013-03-11 09:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [ab15a99] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-12) [73f4ba93] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-12) [73f4ba93] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [73f4ba93] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 471cdd93 >2013-03-11 09:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [73f4ba93] FINISH, SetVdsStatusVDSCommand, log id: 471cdd93 >2013-03-11 09:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-12) [73f4ba93] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 79fe27b0 >2013-03-11 09:45:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [73f4ba93] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 65d3375d >2013-03-11 09:45:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [73f4ba93] FINISH, GetHardwareInfoVDSCommand, log id: 65d3375d >2013-03-11 09:45:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-12) [3dee364d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:45:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-12) [1e329509] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:45:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-12) [1e329509] FINISH, ActivateVdsVDSCommand, log id: 79fe27b0 >2013-03-11 09:45:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-12) [1e329509] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:45:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-12) [1e329509] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 09:45:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [1e329509] Autorecovering 1 storage domains >2013-03-11 09:45:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-12) [1e329509] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 09:45:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-12) [1e052ef9] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:45:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-12) [1e052ef9] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 9:45 AM >2013-03-11 09:45:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-12) [1e052ef9] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 9:45 AM >2013-03-11 09:45:00,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [1322d614] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: f3e6b87 >2013-03-11 09:45:00,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [1322d614] FINISH, GetHardwareInfoVDSCommand, log id: f3e6b87 >2013-03-11 09:45:00,297 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-75) [6753ae40] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 09:45:00,300 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-75) [28cfb267] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 09:45:00,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7a7a22d3 >2013-03-11 09:45:08,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 09:45:08,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:45:08,595 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 09:45:08,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 7a7a22d3 >2013-03-11 09:45:08,598 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [6395210a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:45:08,612 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6395210a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 9a3f9ce >2013-03-11 09:45:08,612 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [6395210a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 09:45:08,612 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6395210a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 9a3f9ce >2013-03-11 09:45:08,614 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [6395210a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 09:45:08,614 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [28cfb267] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 09:45:08,624 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-75) [6036d3ab] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:45:08,625 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [6036d3ab] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3c7e0782 >2013-03-11 09:45:08,626 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [6036d3ab] FINISH, SetVdsStatusVDSCommand, log id: 3c7e0782 >2013-03-11 09:45:08,632 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-75) [6036d3ab] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 09:45:08,655 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [6e911f4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:45:08,657 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [3d8c7a54] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:45:08,659 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-75) [3d8c7a54] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 09:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [3997f98] Autorecovering 1 hosts >2013-03-11 09:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [3997f98] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [10b50dfd] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [10b50dfd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [10b50dfd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2d52ae24 >2013-03-11 09:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [10b50dfd] FINISH, SetVdsStatusVDSCommand, log id: 2d52ae24 >2013-03-11 09:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-65) [10b50dfd] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4c2df920 >2013-03-11 09:50:00,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [10b50dfd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 21375769 >2013-03-11 09:50:00,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [10b50dfd] FINISH, GetHardwareInfoVDSCommand, log id: 21375769 >2013-03-11 09:50:00,187 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-65) [6bf8baae] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:50:00,190 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-65) [229e74f5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:50:00,191 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-65) [229e74f5] FINISH, ActivateVdsVDSCommand, log id: 4c2df920 >2013-03-11 09:50:00,211 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [229e74f5] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:50:00,212 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-65) [229e74f5] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 09:50:00,213 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [229e74f5] Autorecovering 1 storage domains >2013-03-11 09:50:00,213 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [229e74f5] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 09:50:00,214 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [6fe55d4a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:50:00,214 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [6fe55d4a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 9:50 AM >2013-03-11 09:50:00,216 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [6fe55d4a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 9:50 AM >2013-03-11 09:50:02,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [5cd38640] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5988664b >2013-03-11 09:50:02,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [5cd38640] FINISH, GetHardwareInfoVDSCommand, log id: 5988664b >2013-03-11 09:50:02,936 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-43) [74932ef4] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 09:50:02,939 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-43) [18e09316] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 09:50:02,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 37a72ed >2013-03-11 09:50:10,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 09:50:10,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:50:10,930 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 09:50:10,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 37a72ed >2013-03-11 09:50:10,934 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [60a7a60] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:50:10,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [60a7a60] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7df415f0 >2013-03-11 09:50:10,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [60a7a60] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 09:50:10,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [60a7a60] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7df415f0 >2013-03-11 09:50:10,953 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [18e09316] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 09:50:10,962 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-43) [21e3e70f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:50:10,964 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [21e3e70f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1ce3fb0c >2013-03-11 09:50:10,965 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [21e3e70f] FINISH, SetVdsStatusVDSCommand, log id: 1ce3fb0c >2013-03-11 09:50:10,971 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [21e3e70f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 09:50:10,994 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [78899ddc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:50:10,997 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [3c1a551a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:50:10,998 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-43) [3c1a551a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 09:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [43c55d8a] Autorecovering 1 hosts >2013-03-11 09:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [43c55d8a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [35e2c923] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [35e2c923] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [35e2c923] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 45294f1a >2013-03-11 09:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [35e2c923] FINISH, SetVdsStatusVDSCommand, log id: 45294f1a >2013-03-11 09:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [35e2c923] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 58e807a4 >2013-03-11 09:55:00,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [35e2c923] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2402ec30 >2013-03-11 09:55:00,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [35e2c923] FINISH, GetHardwareInfoVDSCommand, log id: 2402ec30 >2013-03-11 09:55:00,627 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-63) [4b98d83b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:55:00,630 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-63) [e22d9fc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:55:00,631 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [e22d9fc] FINISH, ActivateVdsVDSCommand, log id: 58e807a4 >2013-03-11 09:55:00,650 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [e22d9fc] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 09:55:00,651 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-63) [e22d9fc] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 09:55:00,652 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [e22d9fc] Autorecovering 1 storage domains >2013-03-11 09:55:00,652 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [e22d9fc] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 09:55:00,653 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [4898fbf6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:55:00,653 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [4898fbf6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 9:55 AM >2013-03-11 09:55:00,656 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [4898fbf6] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 9:55 AM >2013-03-11 09:55:02,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [4708f8ed] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 226bf988 >2013-03-11 09:55:02,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [4708f8ed] FINISH, GetHardwareInfoVDSCommand, log id: 226bf988 >2013-03-11 09:55:02,269 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-74) [475a180a] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 09:55:02,273 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-74) [7e060c5d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 09:55:02,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: f2369c7 >2013-03-11 09:55:11,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 09:55:11,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 09:55:11,273 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 09:55:11,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: f2369c7 >2013-03-11 09:55:11,277 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [4eb5fb4e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 09:55:11,291 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [4eb5fb4e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7994fe83 >2013-03-11 09:55:11,291 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [4eb5fb4e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 09:55:11,291 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [4eb5fb4e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7994fe83 >2013-03-11 09:55:11,293 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [4eb5fb4e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 09:55:11,294 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [7e060c5d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 09:55:11,303 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-74) [134ea5af] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:55:11,304 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [134ea5af] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4b4ae75c >2013-03-11 09:55:11,305 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [134ea5af] FINISH, SetVdsStatusVDSCommand, log id: 4b4ae75c >2013-03-11 09:55:11,311 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-74) [134ea5af] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 09:55:11,335 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-74) [3c0055e9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:55:11,338 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-74) [1cea420a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 09:55:11,339 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-74) [1cea420a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 10:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [5dd9492e] Autorecovering 1 hosts >2013-03-11 10:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [5dd9492e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:00:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [690e4bce] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [690e4bce] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:00:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [690e4bce] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 78d0363c >2013-03-11 10:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [690e4bce] FINISH, SetVdsStatusVDSCommand, log id: 78d0363c >2013-03-11 10:00:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [690e4bce] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1f709cab >2013-03-11 10:00:00,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [690e4bce] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2e5ab187 >2013-03-11 10:00:00,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [690e4bce] FINISH, GetHardwareInfoVDSCommand, log id: 2e5ab187 >2013-03-11 10:00:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-69) [6740c940] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:00:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-69) [2d260471] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:00:00,095 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [2d260471] FINISH, ActivateVdsVDSCommand, log id: 1f709cab >2013-03-11 10:00:00,114 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [2d260471] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:00:00,114 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-69) [2d260471] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 10:00:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [2d260471] Autorecovering 1 storage domains >2013-03-11 10:00:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [2d260471] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 10:00:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [4101bdf6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:00:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [4101bdf6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 10:00 AM >2013-03-11 10:00:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [4101bdf6] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 10:00 AM >2013-03-11 10:00:02,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-30) [1874f45f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1fd28961 >2013-03-11 10:00:02,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-30) [1874f45f] FINISH, GetHardwareInfoVDSCommand, log id: 1fd28961 >2013-03-11 10:00:02,615 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-30) [4c8c800a] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 10:00:02,619 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-30) [3229169] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 10:00:02,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 371a0dd0 >2013-03-11 10:00:08,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 10:00:08,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:00:08,662 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 10:00:08,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 371a0dd0 >2013-03-11 10:00:08,666 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [66b691ad] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:00:08,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [66b691ad] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 638d003d >2013-03-11 10:00:08,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [66b691ad] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 10:00:08,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [66b691ad] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 638d003d >2013-03-11 10:00:08,682 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [66b691ad] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 10:00:08,682 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [3229169] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 10:00:08,692 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-30) [43bc94db] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:00:08,693 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-30) [43bc94db] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 314a1bd6 >2013-03-11 10:00:08,695 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-30) [43bc94db] FINISH, SetVdsStatusVDSCommand, log id: 314a1bd6 >2013-03-11 10:00:08,700 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-30) [43bc94db] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 10:00:08,723 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-30) [2231b6be] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:00:08,726 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-30) [4c4f58bc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:00:08,727 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-30) [4c4f58bc] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 10:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [5e18beab] Autorecovering 1 hosts >2013-03-11 10:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [5e18beab] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [4d19374a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [4d19374a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [4d19374a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7abd813b >2013-03-11 10:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [4d19374a] FINISH, SetVdsStatusVDSCommand, log id: 7abd813b >2013-03-11 10:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [4d19374a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6f957e84 >2013-03-11 10:05:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [4d19374a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 556f9342 >2013-03-11 10:05:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [4d19374a] FINISH, GetHardwareInfoVDSCommand, log id: 556f9342 >2013-03-11 10:05:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-73) [702d3c6e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:05:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-73) [3e08eb08] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:05:00,200 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [3e08eb08] FINISH, ActivateVdsVDSCommand, log id: 6f957e84 >2013-03-11 10:05:00,219 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [3e08eb08] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:05:00,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-73) [3e08eb08] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 10:05:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [3e08eb08] Autorecovering 1 storage domains >2013-03-11 10:05:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [3e08eb08] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 10:05:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [42b35db9] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:05:00,222 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [42b35db9] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 10:05 AM >2013-03-11 10:05:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [42b35db9] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 10:05 AM >2013-03-11 10:05:02,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [6fe55d4a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 149e752d >2013-03-11 10:05:02,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [6fe55d4a] FINISH, GetHardwareInfoVDSCommand, log id: 149e752d >2013-03-11 10:05:03,000 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-65) [d0fa1b2] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 10:05:03,003 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-65) [52409705] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 10:05:03,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6bc3ba9a >2013-03-11 10:05:09,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 10:05:09,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:05:09,997 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 10:05:09,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 6bc3ba9a >2013-03-11 10:05:10,001 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [c1ec5fb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:05:10,014 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [c1ec5fb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 76b93390 >2013-03-11 10:05:10,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [c1ec5fb] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 10:05:10,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [c1ec5fb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 76b93390 >2013-03-11 10:05:10,017 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [c1ec5fb] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 10:05:10,017 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [52409705] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 10:05:10,026 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-65) [70a485cf] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:05:10,028 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [70a485cf] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: e84565e >2013-03-11 10:05:10,029 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [70a485cf] FINISH, SetVdsStatusVDSCommand, log id: e84565e >2013-03-11 10:05:10,035 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-65) [70a485cf] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 10:05:10,062 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-65) [3343a87e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:05:10,065 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-65) [430467a3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:05:10,066 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-65) [430467a3] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 10:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [5e033ff9] Autorecovering 1 hosts >2013-03-11 10:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [5e033ff9] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [69227237] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [69227237] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:10:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [69227237] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6c7b86cc >2013-03-11 10:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [69227237] FINISH, SetVdsStatusVDSCommand, log id: 6c7b86cc >2013-03-11 10:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-1) [69227237] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2f0acbd2 >2013-03-11 10:10:00,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [69227237] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 249278e0 >2013-03-11 10:10:00,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [69227237] FINISH, GetHardwareInfoVDSCommand, log id: 249278e0 >2013-03-11 10:10:00,319 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-1) [75953318] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:10:00,322 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-1) [1adbf022] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:10:00,323 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-1) [1adbf022] FINISH, ActivateVdsVDSCommand, log id: 2f0acbd2 >2013-03-11 10:10:00,343 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [1adbf022] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:10:00,344 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-1) [1adbf022] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 10:10:00,345 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [1adbf022] Autorecovering 1 storage domains >2013-03-11 10:10:00,345 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [1adbf022] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 10:10:00,346 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [3f877446] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:10:00,346 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [3f877446] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 10:10 AM >2013-03-11 10:10:00,349 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [3f877446] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 10:10 AM >2013-03-11 10:10:01,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [1bfec205] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 10c140ab >2013-03-11 10:10:01,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [1bfec205] FINISH, GetHardwareInfoVDSCommand, log id: 10c140ab >2013-03-11 10:10:01,340 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-27) [3ee8c92d] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 10:10:01,344 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-27) [4fd68af2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 10:10:01,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3cd7d089 >2013-03-11 10:10:08,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 10:10:08,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:10:08,354 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 10:10:08,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 3cd7d089 >2013-03-11 10:10:08,357 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [501bdac] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:10:08,371 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [501bdac] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6d506fa7 >2013-03-11 10:10:08,371 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [501bdac] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 10:10:08,371 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [501bdac] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6d506fa7 >2013-03-11 10:10:08,373 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [501bdac] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 10:10:08,373 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [4fd68af2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 10:10:08,383 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-27) [70ee34ca] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:10:08,384 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [70ee34ca] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 301646a1 >2013-03-11 10:10:08,386 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [70ee34ca] FINISH, SetVdsStatusVDSCommand, log id: 301646a1 >2013-03-11 10:10:08,391 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-27) [70ee34ca] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 10:10:08,420 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-27) [44387c22] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:10:08,423 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-27) [7fb2ac54] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:10:08,424 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-27) [7fb2ac54] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 10:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [47ada7d0] Autorecovering 1 hosts >2013-03-11 10:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [47ada7d0] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [33c994b6] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [33c994b6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [33c994b6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5781a335 >2013-03-11 10:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [33c994b6] FINISH, SetVdsStatusVDSCommand, log id: 5781a335 >2013-03-11 10:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [33c994b6] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5dcfcd8a >2013-03-11 10:15:00,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [33c994b6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: e400f04 >2013-03-11 10:15:00,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [33c994b6] FINISH, GetHardwareInfoVDSCommand, log id: e400f04 >2013-03-11 10:15:00,258 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-96) [6b218fac] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:15:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-96) [35a81fec] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:15:00,262 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [35a81fec] FINISH, ActivateVdsVDSCommand, log id: 5dcfcd8a >2013-03-11 10:15:00,281 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [35a81fec] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:15:00,282 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-96) [35a81fec] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 10:15:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [35a81fec] Autorecovering 1 storage domains >2013-03-11 10:15:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [35a81fec] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 10:15:00,283 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [ca750b4] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:15:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [ca750b4] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 10:15 AM >2013-03-11 10:15:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [ca750b4] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 10:15 AM >2013-03-11 10:15:02,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [5142ccc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 658216bf >2013-03-11 10:15:02,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [5142ccc] FINISH, GetHardwareInfoVDSCommand, log id: 658216bf >2013-03-11 10:15:02,700 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-60) [40e9749e] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 10:15:02,703 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-60) [340a5afd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 10:15:02,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5747389c >2013-03-11 10:15:08,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 10:15:08,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:15:08,695 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 10:15:08,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 5747389c >2013-03-11 10:15:08,699 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [266d8f96] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:15:08,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [266d8f96] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6814865d >2013-03-11 10:15:08,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [266d8f96] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 10:15:08,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [266d8f96] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6814865d >2013-03-11 10:15:08,714 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [266d8f96] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 10:15:08,715 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [340a5afd] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 10:15:08,725 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-60) [6d29daf1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:15:08,727 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [6d29daf1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 43d299cc >2013-03-11 10:15:08,728 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [6d29daf1] FINISH, SetVdsStatusVDSCommand, log id: 43d299cc >2013-03-11 10:15:08,734 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-60) [6d29daf1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 10:15:08,757 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-60) [30d58980] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:15:08,760 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-60) [2121b38e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:15:08,761 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-60) [2121b38e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 10:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [56ad64a3] Autorecovering 1 hosts >2013-03-11 10:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [56ad64a3] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [655fa433] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [655fa433] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [655fa433] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 39146056 >2013-03-11 10:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [655fa433] FINISH, SetVdsStatusVDSCommand, log id: 39146056 >2013-03-11 10:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [655fa433] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7a735f8f >2013-03-11 10:20:00,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [655fa433] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: aaddb5b >2013-03-11 10:20:00,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [655fa433] FINISH, GetHardwareInfoVDSCommand, log id: aaddb5b >2013-03-11 10:20:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [2a86d3ef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:20:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [1cd6e185] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:20:00,205 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [1cd6e185] FINISH, ActivateVdsVDSCommand, log id: 7a735f8f >2013-03-11 10:20:00,224 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [1cd6e185] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:20:00,225 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [1cd6e185] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 10:20:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [1cd6e185] Autorecovering 1 storage domains >2013-03-11 10:20:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [1cd6e185] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 10:20:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [16cdcc0e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:20:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [16cdcc0e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 10:20 AM >2013-03-11 10:20:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [16cdcc0e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 10:20 AM >2013-03-11 10:20:03,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [2b0dfecd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6583f56e >2013-03-11 10:20:03,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [2b0dfecd] FINISH, GetHardwareInfoVDSCommand, log id: 6583f56e >2013-03-11 10:20:03,078 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-41) [349c425f] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 10:20:03,081 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-41) [72410a2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 10:20:03,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7203e9b4 >2013-03-11 10:20:11,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 10:20:11,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:20:11,055 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 10:20:11,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 7203e9b4 >2013-03-11 10:20:11,058 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [549b2709] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:20:11,072 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [549b2709] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7af2225a >2013-03-11 10:20:11,072 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [549b2709] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 10:20:11,072 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [549b2709] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7af2225a >2013-03-11 10:20:11,074 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [549b2709] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 10:20:11,074 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [72410a2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 10:20:11,084 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-41) [6fbab7d4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:20:11,085 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [6fbab7d4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5550c3b7 >2013-03-11 10:20:11,087 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [6fbab7d4] FINISH, SetVdsStatusVDSCommand, log id: 5550c3b7 >2013-03-11 10:20:11,093 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-41) [6fbab7d4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 10:20:11,116 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-41) [72cb891c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:20:11,119 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-41) [6d22a015] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:20:11,120 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-41) [6d22a015] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 10:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [572158f7] Autorecovering 1 hosts >2013-03-11 10:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [572158f7] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [51347b20] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [51347b20] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [51347b20] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7f52a221 >2013-03-11 10:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [51347b20] FINISH, SetVdsStatusVDSCommand, log id: 7f52a221 >2013-03-11 10:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [51347b20] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4b9e7058 >2013-03-11 10:25:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [51347b20] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: e1316d1 >2013-03-11 10:25:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [51347b20] FINISH, GetHardwareInfoVDSCommand, log id: e1316d1 >2013-03-11 10:25:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [394e5447] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:25:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [690a8100] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:25:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [690a8100] FINISH, ActivateVdsVDSCommand, log id: 4b9e7058 >2013-03-11 10:25:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [690a8100] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:25:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [690a8100] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 10:25:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [690a8100] Autorecovering 1 storage domains >2013-03-11 10:25:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [690a8100] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 10:25:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [4d7ca9f7] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:25:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [4d7ca9f7] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 10:25 AM >2013-03-11 10:25:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [4d7ca9f7] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 10:25 AM >2013-03-11 10:25:02,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [3f877446] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 27e81fc3 >2013-03-11 10:25:02,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [3f877446] FINISH, GetHardwareInfoVDSCommand, log id: 27e81fc3 >2013-03-11 10:25:02,396 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-1) [6e274656] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 10:25:02,400 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-1) [9762b06] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 10:25:02,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 380d6e5e >2013-03-11 10:25:09,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 10:25:09,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:25:09,734 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 10:25:09,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 380d6e5e >2013-03-11 10:25:09,738 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [30f4d720] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:25:09,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [30f4d720] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 469fe956 >2013-03-11 10:25:09,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [30f4d720] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 10:25:09,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [30f4d720] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 469fe956 >2013-03-11 10:25:09,753 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [30f4d720] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 10:25:09,754 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [9762b06] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 10:25:09,763 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-1) [72c21f98] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:25:09,765 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [72c21f98] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 199b3675 >2013-03-11 10:25:09,766 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [72c21f98] FINISH, SetVdsStatusVDSCommand, log id: 199b3675 >2013-03-11 10:25:09,772 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-1) [72c21f98] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 10:25:09,795 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-1) [719a4ea3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:25:09,798 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-1) [664bb90] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:25:09,799 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-1) [664bb90] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 10:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [eafeceb] Autorecovering 1 hosts >2013-03-11 10:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [eafeceb] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-95) [1b95319e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-95) [1b95319e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:30:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [1b95319e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3bbe5318 >2013-03-11 10:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [1b95319e] FINISH, SetVdsStatusVDSCommand, log id: 3bbe5318 >2013-03-11 10:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-95) [1b95319e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 67809bc4 >2013-03-11 10:30:00,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [1b95319e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 66969d1 >2013-03-11 10:30:00,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [1b95319e] FINISH, GetHardwareInfoVDSCommand, log id: 66969d1 >2013-03-11 10:30:00,311 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-95) [3c273208] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:30:00,313 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-95) [62e38d28] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:30:00,315 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-95) [62e38d28] FINISH, ActivateVdsVDSCommand, log id: 67809bc4 >2013-03-11 10:30:00,366 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-95) [62e38d28] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:30:00,367 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-95) [62e38d28] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 10:30:00,368 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [62e38d28] Autorecovering 1 storage domains >2013-03-11 10:30:00,368 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-95) [62e38d28] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 10:30:00,369 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-95) [9dcaf51] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:30:00,369 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-95) [9dcaf51] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 10:30 AM >2013-03-11 10:30:00,372 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-95) [9dcaf51] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 10:30 AM >2013-03-11 10:30:01,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [12f4bb98] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: c5a45f1 >2013-03-11 10:30:01,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [12f4bb98] FINISH, GetHardwareInfoVDSCommand, log id: c5a45f1 >2013-03-11 10:30:01,079 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-37) [72152828] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 10:30:01,082 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-37) [483a4da5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 10:30:01,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1e554165 >2013-03-11 10:30:09,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 10:30:09,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:30:09,126 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 10:30:09,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 1e554165 >2013-03-11 10:30:09,130 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [5e034a49] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:30:09,144 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [5e034a49] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1d80f72c >2013-03-11 10:30:09,144 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [5e034a49] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 10:30:09,144 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [5e034a49] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1d80f72c >2013-03-11 10:30:09,146 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [5e034a49] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 10:30:09,146 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [483a4da5] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 10:30:09,157 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-37) [65bbcfde] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:30:09,158 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [65bbcfde] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1502ba93 >2013-03-11 10:30:09,159 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [65bbcfde] FINISH, SetVdsStatusVDSCommand, log id: 1502ba93 >2013-03-11 10:30:09,165 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-37) [65bbcfde] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 10:30:09,188 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-37) [63dc5c5a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:30:09,190 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-37) [40ec6559] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:30:09,192 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-37) [40ec6559] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 10:30:33,881 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-55) [5590818f] Start refreshing all users data >2013-03-11 10:30:35,074 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-63) [4898fbf6] Attempting to update VMs/Templates Ovf. >2013-03-11 10:30:35,075 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-63) [4898fbf6] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 10:30:35,076 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-63) [4898fbf6] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 10:30:35,076 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-63) [4898fbf6] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 10:30:35,077 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-63) [4898fbf6] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 10:30:35,077 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-63) [4898fbf6] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 10:30:35,077 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-63) [4898fbf6] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 10:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [4c69db44] Autorecovering 1 hosts >2013-03-11 10:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [4c69db44] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [19a60e8e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [19a60e8e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [19a60e8e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 68785a0a >2013-03-11 10:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [19a60e8e] FINISH, SetVdsStatusVDSCommand, log id: 68785a0a >2013-03-11 10:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-22) [19a60e8e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 44119bc5 >2013-03-11 10:35:00,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [19a60e8e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 469cddc6 >2013-03-11 10:35:00,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [19a60e8e] FINISH, GetHardwareInfoVDSCommand, log id: 469cddc6 >2013-03-11 10:35:00,259 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [6b9833ee] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:35:00,266 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [22f2b46b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:35:00,268 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-22) [22f2b46b] FINISH, ActivateVdsVDSCommand, log id: 44119bc5 >2013-03-11 10:35:00,287 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [22f2b46b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:35:00,288 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [22f2b46b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 10:35:00,289 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [22f2b46b] Autorecovering 1 storage domains >2013-03-11 10:35:00,289 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [22f2b46b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 10:35:00,290 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [7fa5e9ad] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:35:00,290 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [7fa5e9ad] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 10:35 AM >2013-03-11 10:35:00,293 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [7fa5e9ad] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 10:35 AM >2013-03-11 10:35:00,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [5e21f447] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4cc47d4 >2013-03-11 10:35:00,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [5e21f447] FINISH, GetHardwareInfoVDSCommand, log id: 4cc47d4 >2013-03-11 10:35:00,475 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-18) [3f3963be] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 10:35:00,479 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-18) [7efaa035] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 10:35:00,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5e8e1af8 >2013-03-11 10:35:04,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 10:35:04,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:35:04,775 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 10:35:04,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 5e8e1af8 >2013-03-11 10:35:04,779 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [294ee850] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:35:04,792 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [294ee850] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5e566115 >2013-03-11 10:35:04,792 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [294ee850] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 10:35:04,792 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [294ee850] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5e566115 >2013-03-11 10:35:04,794 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [294ee850] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 10:35:04,794 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [7efaa035] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 10:35:04,804 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-18) [2799e5b9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:35:04,805 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [2799e5b9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 54b74b68 >2013-03-11 10:35:04,807 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [2799e5b9] FINISH, SetVdsStatusVDSCommand, log id: 54b74b68 >2013-03-11 10:35:04,813 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [2799e5b9] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 10:35:04,837 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [283b4d26] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:35:04,839 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [34c31a45] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:35:04,841 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) [34c31a45] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 10:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [4d7ca9f7] Autorecovering 1 hosts >2013-03-11 10:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [4d7ca9f7] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [2a8ca00] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [2a8ca00] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [2a8ca00] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7a347688 >2013-03-11 10:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [2a8ca00] FINISH, SetVdsStatusVDSCommand, log id: 7a347688 >2013-03-11 10:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [2a8ca00] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6da77a8f >2013-03-11 10:40:00,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [2a8ca00] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5f9be23f >2013-03-11 10:40:00,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [2a8ca00] FINISH, GetHardwareInfoVDSCommand, log id: 5f9be23f >2013-03-11 10:40:00,212 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [6101831f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:40:00,214 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [acc578c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:40:00,215 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-26) [acc578c] FINISH, ActivateVdsVDSCommand, log id: 6da77a8f >2013-03-11 10:40:00,235 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-26) [acc578c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:40:00,235 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [acc578c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 10:40:00,237 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [acc578c] Autorecovering 1 storage domains >2013-03-11 10:40:00,237 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-26) [acc578c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 10:40:00,237 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [296830f7] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:40:00,237 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [296830f7] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 10:40 AM >2013-03-11 10:40:00,240 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-26) [296830f7] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 10:40 AM >2013-03-11 10:40:02,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [1d4ef37d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 79d8d8ed >2013-03-11 10:40:02,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [1d4ef37d] FINISH, GetHardwareInfoVDSCommand, log id: 79d8d8ed >2013-03-11 10:40:02,124 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-38) [22599fc0] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 10:40:02,128 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-38) [1b800b36] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 10:40:02,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 8a6d93e >2013-03-11 10:40:08,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 10:40:08,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:40:08,126 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 10:40:08,126 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 8a6d93e >2013-03-11 10:40:08,130 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [66f73631] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:40:08,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [66f73631] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5952aff4 >2013-03-11 10:40:08,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [66f73631] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 10:40:08,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [66f73631] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5952aff4 >2013-03-11 10:40:08,149 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [66f73631] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 10:40:08,150 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [1b800b36] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 10:40:08,178 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-38) [7b70b5c6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:40:08,179 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [7b70b5c6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 67a85861 >2013-03-11 10:40:08,181 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [7b70b5c6] FINISH, SetVdsStatusVDSCommand, log id: 67a85861 >2013-03-11 10:40:08,199 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-38) [7b70b5c6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 10:40:08,227 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-38) [9dd41ec] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:40:08,230 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-38) [58bc9d93] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:40:08,231 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-38) [58bc9d93] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 10:45:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [7991bad5] Autorecovering 1 hosts >2013-03-11 10:45:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [7991bad5] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:45:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-5) [6025340d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:45:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-5) [6025340d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [6025340d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1dcb6628 >2013-03-11 10:45:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [6025340d] FINISH, SetVdsStatusVDSCommand, log id: 1dcb6628 >2013-03-11 10:45:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-5) [6025340d] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4c2c2930 >2013-03-11 10:45:00,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [6025340d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 51a9f9da >2013-03-11 10:45:00,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [6025340d] FINISH, GetHardwareInfoVDSCommand, log id: 51a9f9da >2013-03-11 10:45:00,210 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-5) [741a79be] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:45:00,213 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-5) [582595f6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:45:00,214 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-5) [582595f6] FINISH, ActivateVdsVDSCommand, log id: 4c2c2930 >2013-03-11 10:45:00,237 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-5) [582595f6] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:45:00,238 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-5) [582595f6] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 10:45:00,240 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [582595f6] Autorecovering 1 storage domains >2013-03-11 10:45:00,240 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [582595f6] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 10:45:00,241 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-5) [ecf26e7] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:45:00,241 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-5) [ecf26e7] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 10:45 AM >2013-03-11 10:45:00,247 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-5) [ecf26e7] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 10:45 AM >2013-03-11 10:45:02,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [ca750b4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 454a0f72 >2013-03-11 10:45:02,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [ca750b4] FINISH, GetHardwareInfoVDSCommand, log id: 454a0f72 >2013-03-11 10:45:02,517 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-96) [5886e995] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 10:45:02,520 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-96) [1905cef9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 10:45:02,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 78f3f219 >2013-03-11 10:45:10,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 10:45:10,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:45:10,509 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 10:45:10,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 78f3f219 >2013-03-11 10:45:10,513 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [62e66063] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:45:10,528 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [62e66063] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5c1c6ac4 >2013-03-11 10:45:10,528 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [62e66063] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 10:45:10,528 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [62e66063] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5c1c6ac4 >2013-03-11 10:45:10,530 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [62e66063] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 10:45:10,531 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [1905cef9] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 10:45:10,542 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-96) [22904589] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:45:10,544 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [22904589] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7c5e22ec >2013-03-11 10:45:10,545 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [22904589] FINISH, SetVdsStatusVDSCommand, log id: 7c5e22ec >2013-03-11 10:45:10,551 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-96) [22904589] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 10:45:10,575 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-96) [47a1f255] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:45:10,577 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-96) [1bfd15b9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:45:10,578 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-96) [1bfd15b9] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 10:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [137d1c0a] Autorecovering 1 hosts >2013-03-11 10:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [137d1c0a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [30394288] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [30394288] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [30394288] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: cc408f9 >2013-03-11 10:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-24) [30394288] FINISH, SetVdsStatusVDSCommand, log id: cc408f9 >2013-03-11 10:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-24) [30394288] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 974103b >2013-03-11 10:50:00,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [30394288] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6865057e >2013-03-11 10:50:00,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-24) [30394288] FINISH, GetHardwareInfoVDSCommand, log id: 6865057e >2013-03-11 10:50:00,598 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-24) [6709cfef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:50:00,600 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-24) [755c5956] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:50:00,602 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-24) [755c5956] FINISH, ActivateVdsVDSCommand, log id: 974103b >2013-03-11 10:50:00,621 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-24) [755c5956] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:50:00,622 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-24) [755c5956] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 10:50:00,624 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [755c5956] Autorecovering 1 storage domains >2013-03-11 10:50:00,624 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-24) [755c5956] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 10:50:00,624 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [3df205b5] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:50:00,624 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [3df205b5] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 10:50 AM >2013-03-11 10:50:00,627 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-24) [3df205b5] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 10:50 AM >2013-03-11 10:50:01,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [7fa5e9ad] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 442f9b7e >2013-03-11 10:50:01,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [7fa5e9ad] FINISH, GetHardwareInfoVDSCommand, log id: 442f9b7e >2013-03-11 10:50:01,854 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-22) [50e16f96] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 10:50:01,858 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-22) [5cd4ab2e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 10:50:01,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 30f86f5a >2013-03-11 10:50:08,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 10:50:08,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:50:08,864 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 10:50:08,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 30f86f5a >2013-03-11 10:50:08,868 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [121f72a8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:50:08,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [121f72a8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 61c2826a >2013-03-11 10:50:08,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [121f72a8] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 10:50:08,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [121f72a8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 61c2826a >2013-03-11 10:50:08,884 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [121f72a8] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 10:50:08,884 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [5cd4ab2e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 10:50:08,900 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-22) [47a17d66] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:50:08,902 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [47a17d66] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2bb782cd >2013-03-11 10:50:08,903 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [47a17d66] FINISH, SetVdsStatusVDSCommand, log id: 2bb782cd >2013-03-11 10:50:08,910 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [47a17d66] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 10:50:08,955 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [536cd78a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:50:08,960 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [69312670] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:50:08,962 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-22) [69312670] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 10:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [1410c527] Autorecovering 1 hosts >2013-03-11 10:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [1410c527] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [30fd2c17] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [30fd2c17] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [30fd2c17] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2eaa9fa >2013-03-11 10:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [30fd2c17] FINISH, SetVdsStatusVDSCommand, log id: 2eaa9fa >2013-03-11 10:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [30fd2c17] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 46575c65 >2013-03-11 10:55:00,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [30fd2c17] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 17291084 >2013-03-11 10:55:00,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [30fd2c17] FINISH, GetHardwareInfoVDSCommand, log id: 17291084 >2013-03-11 10:55:00,087 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-45) [18974638] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:55:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-45) [41e91d2e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:55:00,090 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-45) [41e91d2e] FINISH, ActivateVdsVDSCommand, log id: 46575c65 >2013-03-11 10:55:00,110 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-45) [41e91d2e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 10:55:00,111 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-45) [41e91d2e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 10:55:00,112 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [41e91d2e] Autorecovering 1 storage domains >2013-03-11 10:55:00,112 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-45) [41e91d2e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 10:55:00,113 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [5bb73d9b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:55:00,113 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [5bb73d9b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 10:55 AM >2013-03-11 10:55:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-45) [5bb73d9b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 10:55 AM >2013-03-11 10:55:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [e79098a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 73f07a7b >2013-03-11 10:55:00,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [e79098a] FINISH, GetHardwareInfoVDSCommand, log id: 73f07a7b >2013-03-11 10:55:00,241 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-84) [612a458c] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 10:55:00,244 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-84) [112958c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 10:55:00,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3d2b094b >2013-03-11 10:55:05,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 10:55:05,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 10:55:05,226 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 10:55:05,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 3d2b094b >2013-03-11 10:55:05,229 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [322ed0d5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 10:55:05,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [322ed0d5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 30c9e3c9 >2013-03-11 10:55:05,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [322ed0d5] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 10:55:05,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [322ed0d5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 30c9e3c9 >2013-03-11 10:55:05,248 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [112958c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 10:55:05,258 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-84) [38f3e91c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:55:05,259 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [38f3e91c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 594541e >2013-03-11 10:55:05,261 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [38f3e91c] FINISH, SetVdsStatusVDSCommand, log id: 594541e >2013-03-11 10:55:05,266 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-84) [38f3e91c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 10:55:05,290 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-84) [4d47e6e4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:55:05,292 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-84) [1b0f8e9f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 10:55:05,294 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-84) [1b0f8e9f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 11:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [d05bffd] Autorecovering 1 hosts >2013-03-11 11:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [d05bffd] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [fc64934] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [fc64934] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [fc64934] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 87a32d9 >2013-03-11 11:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [fc64934] FINISH, SetVdsStatusVDSCommand, log id: 87a32d9 >2013-03-11 11:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-2) [fc64934] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2213472b >2013-03-11 11:00:00,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [fc64934] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 339adba2 >2013-03-11 11:00:00,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [fc64934] FINISH, GetHardwareInfoVDSCommand, log id: 339adba2 >2013-03-11 11:00:00,103 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [1c91f4cc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:00:00,106 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [4290eccf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:00:00,107 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-2) [4290eccf] FINISH, ActivateVdsVDSCommand, log id: 2213472b >2013-03-11 11:00:00,126 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-2) [4290eccf] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:00:00,127 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [4290eccf] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 11:00:00,129 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [4290eccf] Autorecovering 1 storage domains >2013-03-11 11:00:00,129 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-2) [4290eccf] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 11:00:00,129 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [3fb670c3] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:00:00,129 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [3fb670c3] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 11:00 AM >2013-03-11 11:00:00,132 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-2) [3fb670c3] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 11:00 AM >2013-03-11 11:00:02,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [9dcaf51] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4bfb570a >2013-03-11 11:00:02,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [9dcaf51] FINISH, GetHardwareInfoVDSCommand, log id: 4bfb570a >2013-03-11 11:00:02,571 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-95) [1d61b6a6] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 11:00:02,575 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-95) [46a69234] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 11:00:02,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 129dd0e9 >2013-03-11 11:00:08,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 11:00:08,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:00:08,642 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 11:00:08,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 129dd0e9 >2013-03-11 11:00:08,646 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [19ca34] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:00:08,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [19ca34] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 15441fc6 >2013-03-11 11:00:08,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [19ca34] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 11:00:08,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [19ca34] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 15441fc6 >2013-03-11 11:00:08,662 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [19ca34] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 11:00:08,662 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [46a69234] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 11:00:08,671 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-95) [578a1ad5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:00:08,672 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [578a1ad5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 478fb3df >2013-03-11 11:00:08,674 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [578a1ad5] FINISH, SetVdsStatusVDSCommand, log id: 478fb3df >2013-03-11 11:00:08,679 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-95) [578a1ad5] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 11:00:08,702 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-95) [20374c5d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:00:08,705 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-95) [5f47c13d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:00:08,706 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-95) [5f47c13d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 11:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [1fdb2f9b] Autorecovering 1 hosts >2013-03-11 11:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [1fdb2f9b] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [1c644487] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [1c644487] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [1c644487] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7573b35 >2013-03-11 11:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-78) [1c644487] FINISH, SetVdsStatusVDSCommand, log id: 7573b35 >2013-03-11 11:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-78) [1c644487] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 133e3c91 >2013-03-11 11:05:00,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [1c644487] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 72ed9512 >2013-03-11 11:05:00,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-78) [1c644487] FINISH, GetHardwareInfoVDSCommand, log id: 72ed9512 >2013-03-11 11:05:00,086 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-78) [6b7eae3d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:05:00,089 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-78) [3e5b146e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:05:00,090 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-78) [3e5b146e] FINISH, ActivateVdsVDSCommand, log id: 133e3c91 >2013-03-11 11:05:00,109 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-78) [3e5b146e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:05:00,110 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-78) [3e5b146e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 11:05:00,111 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [3e5b146e] Autorecovering 1 storage domains >2013-03-11 11:05:00,112 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-78) [3e5b146e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 11:05:00,112 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [2312340a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:05:00,112 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [2312340a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 11:05 AM >2013-03-11 11:05:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-78) [2312340a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 11:05 AM >2013-03-11 11:05:02,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [6b031400] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1e17d9c0 >2013-03-11 11:05:02,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [6b031400] FINISH, GetHardwareInfoVDSCommand, log id: 1e17d9c0 >2013-03-11 11:05:02,991 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-76) [7cb0096f] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 11:05:02,994 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-76) [55a1d84e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 11:05:03,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 402a83c1 >2013-03-11 11:05:10,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 11:05:10,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:05:10,003 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 11:05:10,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 402a83c1 >2013-03-11 11:05:10,007 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [347ecb58] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:05:10,021 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [347ecb58] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4360a6a0 >2013-03-11 11:05:10,021 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [347ecb58] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 11:05:10,021 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [347ecb58] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4360a6a0 >2013-03-11 11:05:10,023 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [347ecb58] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 11:05:10,023 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [55a1d84e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 11:05:10,033 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-76) [656b337b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:05:10,034 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [656b337b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 33cc9ed9 >2013-03-11 11:05:10,035 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [656b337b] FINISH, SetVdsStatusVDSCommand, log id: 33cc9ed9 >2013-03-11 11:05:10,041 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-76) [656b337b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 11:05:10,073 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-76) [58c7f2a0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:05:10,076 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-76) [6dbba9f1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:05:10,077 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-76) [6dbba9f1] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 11:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [5c8317b9] Autorecovering 1 hosts >2013-03-11 11:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [5c8317b9] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-49) [2ab7ad4b] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-49) [2ab7ad4b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:10:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [2ab7ad4b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7251d74b >2013-03-11 11:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [2ab7ad4b] FINISH, SetVdsStatusVDSCommand, log id: 7251d74b >2013-03-11 11:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-49) [2ab7ad4b] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: e90f1f0 >2013-03-11 11:10:00,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [2ab7ad4b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2bc450fe >2013-03-11 11:10:00,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [2ab7ad4b] FINISH, GetHardwareInfoVDSCommand, log id: 2bc450fe >2013-03-11 11:10:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-49) [2089155a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:10:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-49) [31e92cbc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:10:00,094 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-49) [31e92cbc] FINISH, ActivateVdsVDSCommand, log id: e90f1f0 >2013-03-11 11:10:00,114 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-49) [31e92cbc] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:10:00,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-49) [31e92cbc] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 11:10:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [31e92cbc] Autorecovering 1 storage domains >2013-03-11 11:10:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-49) [31e92cbc] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 11:10:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-49) [34f1dce7] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:10:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-49) [34f1dce7] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 11:10 AM >2013-03-11 11:10:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-49) [34f1dce7] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 11:10 AM >2013-03-11 11:10:01,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [5bb73d9b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 77f97636 >2013-03-11 11:10:01,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [5bb73d9b] FINISH, GetHardwareInfoVDSCommand, log id: 77f97636 >2013-03-11 11:10:01,349 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-45) [35b4b411] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 11:10:01,353 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-45) [3764012c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 11:10:01,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2ec4190e >2013-03-11 11:10:07,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 11:10:07,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:10:07,360 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 11:10:07,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2ec4190e >2013-03-11 11:10:07,364 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [49ee7d25] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:10:07,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [49ee7d25] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6258b265 >2013-03-11 11:10:07,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [49ee7d25] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 11:10:07,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [49ee7d25] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6258b265 >2013-03-11 11:10:07,379 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [49ee7d25] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 11:10:07,380 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [3764012c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 11:10:07,389 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-45) [52e646ea] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:10:07,390 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [52e646ea] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2ddd8dd0 >2013-03-11 11:10:07,391 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [52e646ea] FINISH, SetVdsStatusVDSCommand, log id: 2ddd8dd0 >2013-03-11 11:10:07,397 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-45) [52e646ea] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 11:10:07,420 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-45) [2e5a57a4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:10:07,423 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-45) [76c42510] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:10:07,424 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-45) [76c42510] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 11:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [7b4e7db5] Autorecovering 1 hosts >2013-03-11 11:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [7b4e7db5] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [23693f36] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [23693f36] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [23693f36] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2d9dbffc >2013-03-11 11:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [23693f36] FINISH, SetVdsStatusVDSCommand, log id: 2d9dbffc >2013-03-11 11:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [23693f36] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 53f517ba >2013-03-11 11:15:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [23693f36] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 346ebe8b >2013-03-11 11:15:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [23693f36] FINISH, GetHardwareInfoVDSCommand, log id: 346ebe8b >2013-03-11 11:15:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-42) [41526395] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:15:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-42) [31297b9c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:15:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-42) [31297b9c] FINISH, ActivateVdsVDSCommand, log id: 53f517ba >2013-03-11 11:15:00,111 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-42) [31297b9c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:15:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [31297b9c] Autorecovering 1 storage domains >2013-03-11 11:15:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-42) [31297b9c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 11:15:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [3fce8e6e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:15:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [3fce8e6e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 11:15 AM >2013-03-11 11:15:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-42) [3fce8e6e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 11:15 AM >2013-03-11 11:15:01,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [719e370d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 45b3e328 >2013-03-11 11:15:01,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [719e370d] FINISH, GetHardwareInfoVDSCommand, log id: 45b3e328 >2013-03-11 11:15:01,712 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-52) [23e8abb1] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 11:15:01,716 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-52) [37ff2321] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 11:15:01,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1ddb47be >2013-03-11 11:15:07,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 11:15:07,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:15:07,712 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 11:15:07,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 1ddb47be >2013-03-11 11:15:07,716 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [2ef0e8d3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:15:07,729 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [2ef0e8d3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 73b85906 >2013-03-11 11:15:07,730 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [2ef0e8d3] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 11:15:07,730 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [2ef0e8d3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 73b85906 >2013-03-11 11:15:07,732 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [2ef0e8d3] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 11:15:07,732 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [37ff2321] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 11:15:07,742 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-52) [4c5b007b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:15:07,743 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [4c5b007b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5e7b4826 >2013-03-11 11:15:07,745 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [4c5b007b] FINISH, SetVdsStatusVDSCommand, log id: 5e7b4826 >2013-03-11 11:15:07,750 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-52) [4c5b007b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 11:15:07,773 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [507e3eb1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:15:07,775 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [23db8bdb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:15:07,777 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-52) [23db8bdb] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 11:20:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [52915c3] Autorecovering 1 hosts >2013-03-11 11:20:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [52915c3] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-7) [eadd4fb] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-7) [eadd4fb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [eadd4fb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 37492975 >2013-03-11 11:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [eadd4fb] FINISH, SetVdsStatusVDSCommand, log id: 37492975 >2013-03-11 11:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-7) [eadd4fb] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7c2c9132 >2013-03-11 11:20:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [eadd4fb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 23f08f91 >2013-03-11 11:20:00,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [eadd4fb] FINISH, GetHardwareInfoVDSCommand, log id: 23f08f91 >2013-03-11 11:20:00,088 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-7) [19c0a5d4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:20:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-7) [5c76aa41] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:20:00,092 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-7) [5c76aa41] FINISH, ActivateVdsVDSCommand, log id: 7c2c9132 >2013-03-11 11:20:00,112 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-7) [5c76aa41] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:20:00,113 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-7) [5c76aa41] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 11:20:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [5c76aa41] Autorecovering 1 storage domains >2013-03-11 11:20:00,114 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [5c76aa41] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 11:20:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-7) [6c71efc1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:20:00,115 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-7) [6c71efc1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 11:20 AM >2013-03-11 11:20:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-7) [6c71efc1] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 11:20 AM >2013-03-11 11:20:02,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [5560d88a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 126f5a92 >2013-03-11 11:20:02,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-33) [5560d88a] FINISH, GetHardwareInfoVDSCommand, log id: 126f5a92 >2013-03-11 11:20:02,064 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-33) [4f9254b8] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 11:20:02,067 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-33) [30cef19c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 11:20:02,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 512cd45 >2013-03-11 11:20:08,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 11:20:08,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:20:08,069 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 11:20:08,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 512cd45 >2013-03-11 11:20:08,073 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [72f0c9fc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:20:08,087 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [72f0c9fc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7dcbf56c >2013-03-11 11:20:08,087 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [72f0c9fc] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 11:20:08,087 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [72f0c9fc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7dcbf56c >2013-03-11 11:20:08,089 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [72f0c9fc] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 11:20:08,089 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [30cef19c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 11:20:08,099 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-33) [4369b665] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:20:08,100 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [4369b665] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 45770799 >2013-03-11 11:20:08,101 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-33) [4369b665] FINISH, SetVdsStatusVDSCommand, log id: 45770799 >2013-03-11 11:20:08,108 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-33) [4369b665] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 11:20:08,132 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-33) [485be1c9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:20:08,134 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-33) [5e46e855] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:20:08,135 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-33) [5e46e855] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 11:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [65bf0f9] Autorecovering 1 hosts >2013-03-11 11:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [65bf0f9] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-99) [1a9c7e3e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-99) [1a9c7e3e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:25:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [1a9c7e3e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 51b33832 >2013-03-11 11:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [1a9c7e3e] FINISH, SetVdsStatusVDSCommand, log id: 51b33832 >2013-03-11 11:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-99) [1a9c7e3e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6b181f98 >2013-03-11 11:25:00,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [1a9c7e3e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3d354653 >2013-03-11 11:25:00,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [1a9c7e3e] FINISH, GetHardwareInfoVDSCommand, log id: 3d354653 >2013-03-11 11:25:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-99) [6a04934a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:25:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-99) [6b541dbb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:25:00,095 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-99) [6b541dbb] FINISH, ActivateVdsVDSCommand, log id: 6b181f98 >2013-03-11 11:25:00,115 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-99) [6b541dbb] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:25:00,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-99) [6b541dbb] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 11:25:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [6b541dbb] Autorecovering 1 storage domains >2013-03-11 11:25:00,117 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [6b541dbb] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 11:25:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-99) [6cabc5f1] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:25:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-99) [6cabc5f1] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 11:25 AM >2013-03-11 11:25:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-99) [6cabc5f1] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 11:25 AM >2013-03-11 11:25:02,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-29) [1b3d25be] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 39f6fc32 >2013-03-11 11:25:02,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-29) [1b3d25be] FINISH, GetHardwareInfoVDSCommand, log id: 39f6fc32 >2013-03-11 11:25:02,424 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-29) [1a0d1a74] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 11:25:02,427 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-29) [44c8f51d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 11:25:02,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5ca5dd2b >2013-03-11 11:25:09,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 11:25:09,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:25:09,423 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 11:25:09,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 5ca5dd2b >2013-03-11 11:25:09,427 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [28c3da9c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:25:09,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [28c3da9c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7e63847e >2013-03-11 11:25:09,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [28c3da9c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 11:25:09,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [28c3da9c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7e63847e >2013-03-11 11:25:09,444 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [28c3da9c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 11:25:09,445 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [44c8f51d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 11:25:09,456 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-29) [44e98a43] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:25:09,457 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-29) [44e98a43] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 52c18853 >2013-03-11 11:25:09,459 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-29) [44e98a43] FINISH, SetVdsStatusVDSCommand, log id: 52c18853 >2013-03-11 11:25:09,469 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-29) [44e98a43] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 11:25:09,493 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-29) [1273bc4f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:25:09,496 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-29) [7d975ccd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:25:09,498 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-29) [7d975ccd] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 11:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [32a9e514] Autorecovering 1 hosts >2013-03-11 11:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [32a9e514] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [166397c4] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [166397c4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [166397c4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 670ef9a3 >2013-03-11 11:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-89) [166397c4] FINISH, SetVdsStatusVDSCommand, log id: 670ef9a3 >2013-03-11 11:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-89) [166397c4] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 13e7525e >2013-03-11 11:30:00,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [166397c4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 520d9631 >2013-03-11 11:30:00,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [166397c4] FINISH, GetHardwareInfoVDSCommand, log id: 520d9631 >2013-03-11 11:30:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-89) [7e5f9d11] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:30:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-89) [13d0dea3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:30:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-89) [13d0dea3] FINISH, ActivateVdsVDSCommand, log id: 13e7525e >2013-03-11 11:30:00,224 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-89) [13d0dea3] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:30:00,225 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-89) [13d0dea3] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 11:30:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [13d0dea3] Autorecovering 1 storage domains >2013-03-11 11:30:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-89) [13d0dea3] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 11:30:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [1044299d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:30:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [1044299d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 11:30 AM >2013-03-11 11:30:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-89) [1044299d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 11:30 AM >2013-03-11 11:30:00,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [1c5fdbf2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6cab92a9 >2013-03-11 11:30:00,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-90) [1c5fdbf2] FINISH, GetHardwareInfoVDSCommand, log id: 6cab92a9 >2013-03-11 11:30:00,785 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-90) [3401abc3] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 11:30:00,789 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-90) [7331d825] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 11:30:00,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 50dbd8e8 >2013-03-11 11:30:07,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 11:30:07,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:30:07,842 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 11:30:07,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 50dbd8e8 >2013-03-11 11:30:07,846 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [1aace9d9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:30:07,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [1aace9d9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40451dd2 >2013-03-11 11:30:07,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [1aace9d9] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 11:30:07,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [1aace9d9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40451dd2 >2013-03-11 11:30:07,862 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [1aace9d9] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 11:30:07,863 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [7331d825] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 11:30:07,873 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-90) [25c89b4d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:30:07,874 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [25c89b4d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4ab314c7 >2013-03-11 11:30:07,876 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-90) [25c89b4d] FINISH, SetVdsStatusVDSCommand, log id: 4ab314c7 >2013-03-11 11:30:07,881 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-90) [25c89b4d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 11:30:07,904 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-90) [5df23a64] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:30:07,907 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-90) [389116d0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:30:07,908 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-90) [389116d0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 11:30:33,884 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-49) [34f1dce7] Start refreshing all users data >2013-03-11 11:30:35,077 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [7d975ccd] Attempting to update VMs/Templates Ovf. >2013-03-11 11:30:35,078 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [7d975ccd] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 11:30:35,079 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [7d975ccd] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 11:30:35,079 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [7d975ccd] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 11:30:35,080 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [7d975ccd] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 11:30:35,080 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [7d975ccd] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 11:30:35,080 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-29) [7d975ccd] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 11:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [6c71efc1] Autorecovering 1 hosts >2013-03-11 11:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [6c71efc1] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-7) [506b15be] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-7) [506b15be] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [506b15be] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 34217046 >2013-03-11 11:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [506b15be] FINISH, SetVdsStatusVDSCommand, log id: 34217046 >2013-03-11 11:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-7) [506b15be] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7a10423f >2013-03-11 11:35:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [506b15be] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1c216ec0 >2013-03-11 11:35:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [506b15be] FINISH, GetHardwareInfoVDSCommand, log id: 1c216ec0 >2013-03-11 11:35:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-7) [2d618509] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:35:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-7) [4dfa1ea] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:35:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-7) [4dfa1ea] FINISH, ActivateVdsVDSCommand, log id: 7a10423f >2013-03-11 11:35:00,221 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-7) [4dfa1ea] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:35:00,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-7) [4dfa1ea] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 11:35:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [4dfa1ea] Autorecovering 1 storage domains >2013-03-11 11:35:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-7) [4dfa1ea] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 11:35:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-7) [6316c02e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:35:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-7) [6316c02e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 11:35 AM >2013-03-11 11:35:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-7) [6316c02e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 11:35 AM >2013-03-11 11:35:02,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-98) [3365b147] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 18f9757a >2013-03-11 11:35:02,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-98) [3365b147] FINISH, GetHardwareInfoVDSCommand, log id: 18f9757a >2013-03-11 11:35:02,185 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-98) [40311ac7] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 11:35:02,189 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-98) [20b043c0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 11:35:02,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 9e6f3f9 >2013-03-11 11:35:08,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 11:35:08,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:35:08,190 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 11:35:08,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 9e6f3f9 >2013-03-11 11:35:08,194 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [74f2d8be] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:35:08,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [74f2d8be] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2603b2ba >2013-03-11 11:35:08,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [74f2d8be] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 11:35:08,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [74f2d8be] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2603b2ba >2013-03-11 11:35:08,209 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [74f2d8be] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 11:35:08,210 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [20b043c0] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 11:35:08,219 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-98) [3781697a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:35:08,220 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-98) [3781697a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7870ecd4 >2013-03-11 11:35:08,221 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-98) [3781697a] FINISH, SetVdsStatusVDSCommand, log id: 7870ecd4 >2013-03-11 11:35:08,227 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-98) [3781697a] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 11:35:08,251 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-98) [6bfff229] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:35:08,253 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-98) [500d17ae] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:35:08,255 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-98) [500d17ae] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 11:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [6cabc5f1] Autorecovering 1 hosts >2013-03-11 11:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [6cabc5f1] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-99) [7679d063] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-99) [7679d063] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:40:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [7679d063] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6f7d13b4 >2013-03-11 11:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [7679d063] FINISH, SetVdsStatusVDSCommand, log id: 6f7d13b4 >2013-03-11 11:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-99) [7679d063] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 691e59b3 >2013-03-11 11:40:00,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [7679d063] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7b7e513b >2013-03-11 11:40:00,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [7679d063] FINISH, GetHardwareInfoVDSCommand, log id: 7b7e513b >2013-03-11 11:40:00,421 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-99) [7ed9cbab] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:40:00,424 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-99) [154999a1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:40:00,425 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-99) [154999a1] FINISH, ActivateVdsVDSCommand, log id: 691e59b3 >2013-03-11 11:40:00,445 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-99) [154999a1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:40:00,446 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-99) [154999a1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 11:40:00,447 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [154999a1] Autorecovering 1 storage domains >2013-03-11 11:40:00,447 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-99) [154999a1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 11:40:00,448 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-99) [18568a9b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:40:00,448 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-99) [18568a9b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 11:40 AM >2013-03-11 11:40:00,451 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-99) [18568a9b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 11:40 AM >2013-03-11 11:40:02,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [34f1dce7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 14ec2d00 >2013-03-11 11:40:02,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [34f1dce7] FINISH, GetHardwareInfoVDSCommand, log id: 14ec2d00 >2013-03-11 11:40:02,537 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-49) [5ee8dd21] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 11:40:02,541 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-49) [7f8c209e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 11:40:02,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 36aedcba >2013-03-11 11:40:07,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 11:40:07,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:40:07,533 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 11:40:07,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 36aedcba >2013-03-11 11:40:07,537 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [4379b4c6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:40:07,551 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [4379b4c6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5f7966ac >2013-03-11 11:40:07,551 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [4379b4c6] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 11:40:07,551 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [4379b4c6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5f7966ac >2013-03-11 11:40:07,553 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [4379b4c6] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 11:40:07,553 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [7f8c209e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 11:40:07,569 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-49) [7e090b1b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:40:07,570 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [7e090b1b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4dd0c3f9 >2013-03-11 11:40:07,571 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [7e090b1b] FINISH, SetVdsStatusVDSCommand, log id: 4dd0c3f9 >2013-03-11 11:40:07,577 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-49) [7e090b1b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 11:40:07,600 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-49) [a6c5ace] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:40:07,603 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-49) [520adeb3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:40:07,604 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) [520adeb3] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 11:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [706addc6] Autorecovering 1 hosts >2013-03-11 11:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [706addc6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-56) [3c01c897] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-56) [3c01c897] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [3c01c897] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1b7db35d >2013-03-11 11:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [3c01c897] FINISH, SetVdsStatusVDSCommand, log id: 1b7db35d >2013-03-11 11:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-56) [3c01c897] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 286e98de >2013-03-11 11:45:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [3c01c897] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 59f9f2b4 >2013-03-11 11:45:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [3c01c897] FINISH, GetHardwareInfoVDSCommand, log id: 59f9f2b4 >2013-03-11 11:45:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-56) [3aa870b5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:45:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-56) [33f15b4b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:45:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-56) [33f15b4b] FINISH, ActivateVdsVDSCommand, log id: 286e98de >2013-03-11 11:45:00,221 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-56) [33f15b4b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:45:00,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-56) [33f15b4b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 11:45:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [33f15b4b] Autorecovering 1 storage domains >2013-03-11 11:45:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [33f15b4b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 11:45:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-56) [248e1751] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:45:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-56) [248e1751] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 11:45 AM >2013-03-11 11:45:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-56) [248e1751] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 11:45 AM >2013-03-11 11:45:01,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [3fb670c3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3048d82b >2013-03-11 11:45:01,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [3fb670c3] FINISH, GetHardwareInfoVDSCommand, log id: 3048d82b >2013-03-11 11:45:01,882 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-2) [43f7a1bd] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 11:45:01,886 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-2) [7996c1de] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 11:45:01,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 60de6f57 >2013-03-11 11:45:05,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 11:45:05,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:45:05,864 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 11:45:05,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 60de6f57 >2013-03-11 11:45:05,868 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [36d3cb61] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:45:05,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [36d3cb61] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4fc6ea8d >2013-03-11 11:45:05,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [36d3cb61] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 11:45:05,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [36d3cb61] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4fc6ea8d >2013-03-11 11:45:05,883 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [36d3cb61] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 11:45:05,884 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [7996c1de] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 11:45:05,893 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-2) [40ba5de8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:45:05,894 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [40ba5de8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5f700229 >2013-03-11 11:45:05,895 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [40ba5de8] FINISH, SetVdsStatusVDSCommand, log id: 5f700229 >2013-03-11 11:45:05,901 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [40ba5de8] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 11:45:05,924 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [3567176e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:45:05,927 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [1effb5f1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:45:05,928 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-2) [1effb5f1] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 11:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [43ef0f74] Autorecovering 1 hosts >2013-03-11 11:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [43ef0f74] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [6a4dd3fa] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [6a4dd3fa] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [6a4dd3fa] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 38f8a8ef >2013-03-11 11:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [6a4dd3fa] FINISH, SetVdsStatusVDSCommand, log id: 38f8a8ef >2013-03-11 11:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-13) [6a4dd3fa] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2bd2b28f >2013-03-11 11:50:00,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [6a4dd3fa] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 55ac927e >2013-03-11 11:50:00,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [6a4dd3fa] FINISH, GetHardwareInfoVDSCommand, log id: 55ac927e >2013-03-11 11:50:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-13) [601ed1c9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:50:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-13) [137d9a6d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:50:00,205 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-13) [137d9a6d] FINISH, ActivateVdsVDSCommand, log id: 2bd2b28f >2013-03-11 11:50:00,224 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [137d9a6d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:50:00,225 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-13) [137d9a6d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 11:50:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [137d9a6d] Autorecovering 1 storage domains >2013-03-11 11:50:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [137d9a6d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 11:50:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [58279a55] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:50:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [58279a55] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 11:50 AM >2013-03-11 11:50:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [58279a55] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 11:50 AM >2013-03-11 11:50:00,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [514c8dcb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2535c941 >2013-03-11 11:50:00,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [514c8dcb] FINISH, GetHardwareInfoVDSCommand, log id: 2535c941 >2013-03-11 11:50:00,302 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-47) [293f38d4] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 11:50:00,306 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-47) [40c9daa7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 11:50:00,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 30e21beb >2013-03-11 11:50:07,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 11:50:07,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:50:07,296 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 11:50:07,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 30e21beb >2013-03-11 11:50:07,300 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [5494a849] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:50:07,314 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [5494a849] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3397c66 >2013-03-11 11:50:07,314 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [5494a849] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 11:50:07,314 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [5494a849] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3397c66 >2013-03-11 11:50:07,316 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [5494a849] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 11:50:07,316 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [40c9daa7] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 11:50:07,326 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-47) [100eacdd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:50:07,327 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [100eacdd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 782dccd9 >2013-03-11 11:50:07,328 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [100eacdd] FINISH, SetVdsStatusVDSCommand, log id: 782dccd9 >2013-03-11 11:50:07,334 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [100eacdd] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 11:50:07,362 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [6a551f0a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:50:07,364 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [789bca44] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:50:07,366 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-47) [789bca44] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 11:55:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [43f1b61c] Autorecovering 1 hosts >2013-03-11 11:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [43f1b61c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [25fffe7d] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [25fffe7d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [25fffe7d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7c145165 >2013-03-11 11:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [25fffe7d] FINISH, SetVdsStatusVDSCommand, log id: 7c145165 >2013-03-11 11:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-44) [25fffe7d] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 25b9d6be >2013-03-11 11:55:00,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [25fffe7d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 81ea1a >2013-03-11 11:55:00,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [25fffe7d] FINISH, GetHardwareInfoVDSCommand, log id: 81ea1a >2013-03-11 11:55:00,265 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-44) [2135b3f3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:55:00,268 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-44) [3bc77f0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:55:00,269 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-44) [3bc77f0] FINISH, ActivateVdsVDSCommand, log id: 25b9d6be >2013-03-11 11:55:00,289 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-44) [3bc77f0] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 11:55:00,290 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-44) [3bc77f0] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 11:55:00,291 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [3bc77f0] Autorecovering 1 storage domains >2013-03-11 11:55:00,291 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-44) [3bc77f0] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 11:55:00,292 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [13b5f2cd] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:55:00,292 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [13b5f2cd] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 11:55 AM >2013-03-11 11:55:00,295 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-44) [13b5f2cd] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 11:55 AM >2013-03-11 11:55:01,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [20ebd7a8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3aaf0029 >2013-03-11 11:55:01,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-68) [20ebd7a8] FINISH, GetHardwareInfoVDSCommand, log id: 3aaf0029 >2013-03-11 11:55:01,646 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-68) [756b2d57] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 11:55:01,649 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-68) [5ae83de4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 11:55:01,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 184245ef >2013-03-11 11:55:08,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 11:55:08,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 11:55:08,643 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 11:55:08,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 184245ef >2013-03-11 11:55:08,647 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [78b33039] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 11:55:08,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [78b33039] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 781bbbc0 >2013-03-11 11:55:08,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [78b33039] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 11:55:08,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [78b33039] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 781bbbc0 >2013-03-11 11:55:08,666 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [5ae83de4] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 11:55:08,676 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-68) [7e2a9a4e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:55:08,678 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [7e2a9a4e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1e01137 >2013-03-11 11:55:08,679 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-68) [7e2a9a4e] FINISH, SetVdsStatusVDSCommand, log id: 1e01137 >2013-03-11 11:55:08,686 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-68) [7e2a9a4e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 11:55:08,709 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-68) [3a36b8be] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:55:08,712 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-68) [2fda082d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 11:55:08,713 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-68) [2fda082d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 12:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [12bd5363] Autorecovering 1 hosts >2013-03-11 12:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [12bd5363] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-80) [6dd6474f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-80) [6dd6474f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [6dd6474f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 14f321d0 >2013-03-11 12:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [6dd6474f] FINISH, SetVdsStatusVDSCommand, log id: 14f321d0 >2013-03-11 12:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-80) [6dd6474f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1a5aead3 >2013-03-11 12:00:00,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [6dd6474f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1bcca246 >2013-03-11 12:00:00,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [6dd6474f] FINISH, GetHardwareInfoVDSCommand, log id: 1bcca246 >2013-03-11 12:00:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-80) [22a9fe70] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:00:00,207 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-80) [44dc2f78] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:00:00,208 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-80) [44dc2f78] FINISH, ActivateVdsVDSCommand, log id: 1a5aead3 >2013-03-11 12:00:00,227 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-80) [44dc2f78] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:00:00,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-80) [44dc2f78] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 12:00:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [44dc2f78] Autorecovering 1 storage domains >2013-03-11 12:00:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-80) [44dc2f78] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 12:00:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-80) [237fa769] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:00:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-80) [237fa769] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:00 PM >2013-03-11 12:00:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-80) [237fa769] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:00 PM >2013-03-11 12:00:02,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [51ddbdd7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 40617282 >2013-03-11 12:00:02,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [51ddbdd7] FINISH, GetHardwareInfoVDSCommand, log id: 40617282 >2013-03-11 12:00:02,999 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-11) [9c56630] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 12:00:03,003 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-11) [6caeaf5f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 12:00:03,017 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6a467d81 >2013-03-11 12:00:11,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 12:00:11,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:00:11,043 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 12:00:11,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 6a467d81 >2013-03-11 12:00:11,047 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [71b5f105] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:00:11,060 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [71b5f105] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5e935c23 >2013-03-11 12:00:11,061 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [71b5f105] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 12:00:11,061 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [71b5f105] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5e935c23 >2013-03-11 12:00:11,063 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [71b5f105] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 12:00:11,063 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [6caeaf5f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 12:00:11,073 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-11) [6c9d5c9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:00:11,074 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [6c9d5c9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 47052462 >2013-03-11 12:00:11,075 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [6c9d5c9] FINISH, SetVdsStatusVDSCommand, log id: 47052462 >2013-03-11 12:00:11,081 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-11) [6c9d5c9] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 12:00:11,104 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-11) [6237c052] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:00:11,107 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-11) [63789b1d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:00:11,108 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-11) [63789b1d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 12:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-31) [12ddbded] Autorecovering 1 hosts >2013-03-11 12:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-31) [12ddbded] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-31) [133e8ed9] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-31) [133e8ed9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [133e8ed9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 498f1da4 >2013-03-11 12:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-31) [133e8ed9] FINISH, SetVdsStatusVDSCommand, log id: 498f1da4 >2013-03-11 12:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-31) [133e8ed9] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 755a4cf9 >2013-03-11 12:05:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [133e8ed9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 272d2b02 >2013-03-11 12:05:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-31) [133e8ed9] FINISH, GetHardwareInfoVDSCommand, log id: 272d2b02 >2013-03-11 12:05:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-31) [64cee17f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:05:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-31) [6cebbc1f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:05:00,201 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-31) [6cebbc1f] FINISH, ActivateVdsVDSCommand, log id: 755a4cf9 >2013-03-11 12:05:00,220 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-31) [6cebbc1f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:05:00,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-31) [6cebbc1f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 12:05:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-31) [6cebbc1f] Autorecovering 1 storage domains >2013-03-11 12:05:00,222 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-31) [6cebbc1f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 12:05:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-31) [d18b803] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:05:00,223 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-31) [d18b803] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:05 PM >2013-03-11 12:05:00,226 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-31) [d18b803] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:05 PM >2013-03-11 12:05:02,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [6316c02e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 65796fd2 >2013-03-11 12:05:02,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [6316c02e] FINISH, GetHardwareInfoVDSCommand, log id: 65796fd2 >2013-03-11 12:05:02,416 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-7) [1a6001ff] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 12:05:02,419 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-7) [4d8f0e2a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 12:05:02,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: d75fc7f >2013-03-11 12:05:06,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 12:05:06,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:05:06,437 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 12:05:06,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: d75fc7f >2013-03-11 12:05:06,441 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [60678508] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:05:06,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [60678508] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5dacaccb >2013-03-11 12:05:06,455 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [60678508] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 12:05:06,455 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [60678508] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5dacaccb >2013-03-11 12:05:06,457 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [60678508] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 12:05:06,457 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [4d8f0e2a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 12:05:06,466 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-7) [45311ac0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:05:06,468 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [45311ac0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1757f549 >2013-03-11 12:05:06,469 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [45311ac0] FINISH, SetVdsStatusVDSCommand, log id: 1757f549 >2013-03-11 12:05:06,475 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-7) [45311ac0] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 12:05:06,507 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-7) [22193587] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:05:06,509 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-7) [23a8b68f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:05:06,510 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) [23a8b68f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 12:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [3d8c7a54] Autorecovering 1 hosts >2013-03-11 12:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [3d8c7a54] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [3f45f151] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [3f45f151] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:10:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [3f45f151] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6fbf3081 >2013-03-11 12:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [3f45f151] FINISH, SetVdsStatusVDSCommand, log id: 6fbf3081 >2013-03-11 12:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-75) [3f45f151] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 66f066a0 >2013-03-11 12:10:00,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [3f45f151] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5ee5ccbd >2013-03-11 12:10:00,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [3f45f151] FINISH, GetHardwareInfoVDSCommand, log id: 5ee5ccbd >2013-03-11 12:10:00,375 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [ef2282b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:10:00,377 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [1f24183c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:10:00,378 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-75) [1f24183c] FINISH, ActivateVdsVDSCommand, log id: 66f066a0 >2013-03-11 12:10:00,401 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [1f24183c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:10:00,402 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-75) [1f24183c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 12:10:00,403 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [1f24183c] Autorecovering 1 storage domains >2013-03-11 12:10:00,403 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [1f24183c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 12:10:00,404 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [79cd6a09] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:10:00,404 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [79cd6a09] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:10 PM >2013-03-11 12:10:00,407 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [79cd6a09] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:10 PM >2013-03-11 12:10:00,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [90bee01] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: d04b54b >2013-03-11 12:10:00,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [90bee01] FINISH, GetHardwareInfoVDSCommand, log id: d04b54b >2013-03-11 12:10:00,799 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-91) [4bd41429] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 12:10:00,803 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-91) [762756cd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 12:10:00,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: b5ac44e >2013-03-11 12:10:08,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 12:10:08,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:10:08,810 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 12:10:08,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: b5ac44e >2013-03-11 12:10:08,813 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [5aeefcea] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:10:08,827 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [5aeefcea] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 63e5e76 >2013-03-11 12:10:08,827 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [5aeefcea] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 12:10:08,827 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [5aeefcea] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 63e5e76 >2013-03-11 12:10:08,829 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [5aeefcea] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 12:10:08,830 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [762756cd] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 12:10:08,839 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-91) [2b6434c1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:10:08,841 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [2b6434c1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: c0ea0c0 >2013-03-11 12:10:08,842 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [2b6434c1] FINISH, SetVdsStatusVDSCommand, log id: c0ea0c0 >2013-03-11 12:10:08,849 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-91) [2b6434c1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 12:10:08,873 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-91) [332af517] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:10:08,876 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-91) [4f176f90] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:10:08,877 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-91) [4f176f90] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 12:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [3c1a551a] Autorecovering 1 hosts >2013-03-11 12:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [3c1a551a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [7b3c7aa4] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [7b3c7aa4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [7b3c7aa4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 53b68423 >2013-03-11 12:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [7b3c7aa4] FINISH, SetVdsStatusVDSCommand, log id: 53b68423 >2013-03-11 12:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [7b3c7aa4] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1360ea56 >2013-03-11 12:15:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [7b3c7aa4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 360b072a >2013-03-11 12:15:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [7b3c7aa4] FINISH, GetHardwareInfoVDSCommand, log id: 360b072a >2013-03-11 12:15:00,197 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [120e1263] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:15:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [33b1ae6a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:15:00,212 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [33b1ae6a] FINISH, ActivateVdsVDSCommand, log id: 1360ea56 >2013-03-11 12:15:00,232 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [33b1ae6a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:15:00,233 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [33b1ae6a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 12:15:00,235 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [33b1ae6a] Autorecovering 1 storage domains >2013-03-11 12:15:00,235 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [33b1ae6a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 12:15:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [1dc51316] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:15:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [1dc51316] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:15 PM >2013-03-11 12:15:00,238 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [1dc51316] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:15 PM >2013-03-11 12:15:00,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [6dd1154] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5d1a0ff2 >2013-03-11 12:15:00,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [6dd1154] FINISH, GetHardwareInfoVDSCommand, log id: 5d1a0ff2 >2013-03-11 12:15:00,324 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-93) [781cc496] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 12:15:00,327 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-93) [7a603972] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 12:15:00,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3c257cca >2013-03-11 12:15:07,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 12:15:07,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:15:07,663 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 12:15:07,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 3c257cca >2013-03-11 12:15:07,666 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [5c9dca84] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:15:07,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [5c9dca84] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 64a6ee2a >2013-03-11 12:15:07,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [5c9dca84] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 12:15:07,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [5c9dca84] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 64a6ee2a >2013-03-11 12:15:07,684 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [5c9dca84] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 12:15:07,684 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [7a603972] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 12:15:07,694 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-93) [3269d321] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:15:07,696 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [3269d321] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 360c1a5 >2013-03-11 12:15:07,697 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [3269d321] FINISH, SetVdsStatusVDSCommand, log id: 360c1a5 >2013-03-11 12:15:07,702 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-93) [3269d321] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 12:15:07,729 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [69839322] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:15:07,731 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [783825bc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:15:07,733 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-93) [783825bc] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 12:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [5e22dc1b] Autorecovering 1 hosts >2013-03-11 12:20:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [5e22dc1b] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [96af9ff] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [96af9ff] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:20:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [96af9ff] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 60a4c884 >2013-03-11 12:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [96af9ff] FINISH, SetVdsStatusVDSCommand, log id: 60a4c884 >2013-03-11 12:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [96af9ff] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5ad2b37 >2013-03-11 12:20:00,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [96af9ff] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 55801acd >2013-03-11 12:20:00,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [96af9ff] FINISH, GetHardwareInfoVDSCommand, log id: 55801acd >2013-03-11 12:20:00,214 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [40e195d6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:20:00,216 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [167da217] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:20:00,217 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [167da217] FINISH, ActivateVdsVDSCommand, log id: 5ad2b37 >2013-03-11 12:20:00,236 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [167da217] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:20:00,237 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [167da217] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 12:20:00,238 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [167da217] Autorecovering 1 storage domains >2013-03-11 12:20:00,239 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [167da217] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 12:20:00,239 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [44b36dbb] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:20:00,239 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [44b36dbb] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:20 PM >2013-03-11 12:20:00,242 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [44b36dbb] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:20 PM >2013-03-11 12:20:01,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [12380a29] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: a090642 >2013-03-11 12:20:01,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [12380a29] FINISH, GetHardwareInfoVDSCommand, log id: a090642 >2013-03-11 12:20:02,015 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-20) [5b197242] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 12:20:02,018 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-20) [191c3690] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 12:20:02,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7ba31035 >2013-03-11 12:20:10,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 12:20:10,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:20:10,035 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 12:20:10,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 7ba31035 >2013-03-11 12:20:10,039 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [3576c5df] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:20:10,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [3576c5df] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 74c8db3e >2013-03-11 12:20:10,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [3576c5df] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 12:20:10,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [3576c5df] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 74c8db3e >2013-03-11 12:20:10,054 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [3576c5df] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 12:20:10,055 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [191c3690] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 12:20:10,064 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-20) [4b6dc24d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:20:10,065 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [4b6dc24d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 20d3977a >2013-03-11 12:20:10,067 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [4b6dc24d] FINISH, SetVdsStatusVDSCommand, log id: 20d3977a >2013-03-11 12:20:10,072 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-20) [4b6dc24d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 12:20:10,096 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-20) [66e03163] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:20:10,098 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-20) [5e8f40cb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:20:10,100 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-20) [5e8f40cb] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 12:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [6cb94f44] Autorecovering 1 hosts >2013-03-11 12:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [6cb94f44] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [6863070] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [6863070] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:25:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [6863070] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1b090d4d >2013-03-11 12:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [6863070] FINISH, SetVdsStatusVDSCommand, log id: 1b090d4d >2013-03-11 12:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-83) [6863070] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4d59507d >2013-03-11 12:25:00,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [6863070] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 44da57ea >2013-03-11 12:25:00,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [6863070] FINISH, GetHardwareInfoVDSCommand, log id: 44da57ea >2013-03-11 12:25:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-83) [66f11325] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:25:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-83) [12217cd6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:25:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-83) [12217cd6] FINISH, ActivateVdsVDSCommand, log id: 4d59507d >2013-03-11 12:25:00,224 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [12217cd6] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:25:00,225 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-83) [12217cd6] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 12:25:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [12217cd6] Autorecovering 1 storage domains >2013-03-11 12:25:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [12217cd6] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 12:25:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [50a5b579] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:25:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [50a5b579] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:25 PM >2013-03-11 12:25:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [50a5b579] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:25 PM >2013-03-11 12:25:01,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [640da200] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3c742fc6 >2013-03-11 12:25:01,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [640da200] FINISH, GetHardwareInfoVDSCommand, log id: 3c742fc6 >2013-03-11 12:25:01,373 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-48) [5db72fc] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 12:25:01,376 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-48) [7ea56401] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 12:25:01,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4563c7a1 >2013-03-11 12:25:05,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 12:25:05,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:25:05,680 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 12:25:05,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 4563c7a1 >2013-03-11 12:25:05,684 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [72e9a0e9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:25:05,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [72e9a0e9] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 370aa906 >2013-03-11 12:25:05,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [72e9a0e9] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 12:25:05,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [72e9a0e9] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 370aa906 >2013-03-11 12:25:05,700 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [72e9a0e9] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 12:25:05,700 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [7ea56401] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 12:25:05,709 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-48) [173afdfb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:25:05,711 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [173afdfb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1a626384 >2013-03-11 12:25:05,712 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [173afdfb] FINISH, SetVdsStatusVDSCommand, log id: 1a626384 >2013-03-11 12:25:05,718 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-48) [173afdfb] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 12:25:05,741 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-48) [367798da] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:25:05,744 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-48) [5e6e2788] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:25:05,745 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-48) [5e6e2788] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 12:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [50f7ef2f] Autorecovering 1 hosts >2013-03-11 12:30:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [50f7ef2f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:30:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [1f43abec] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [1f43abec] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [1f43abec] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7cabd8bc >2013-03-11 12:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [1f43abec] FINISH, SetVdsStatusVDSCommand, log id: 7cabd8bc >2013-03-11 12:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [1f43abec] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3cf54a7 >2013-03-11 12:30:00,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [1f43abec] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: ef280b2 >2013-03-11 12:30:00,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [1f43abec] FINISH, GetHardwareInfoVDSCommand, log id: ef280b2 >2013-03-11 12:30:00,209 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-79) [5744a62f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:30:00,211 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-79) [29b08a8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:30:00,212 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [29b08a8] FINISH, ActivateVdsVDSCommand, log id: 3cf54a7 >2013-03-11 12:30:00,231 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [29b08a8] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:30:00,232 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-79) [29b08a8] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 12:30:00,233 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [29b08a8] Autorecovering 1 storage domains >2013-03-11 12:30:00,234 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [29b08a8] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 12:30:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [e877d39] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:30:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [e877d39] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:30 PM >2013-03-11 12:30:00,237 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [e877d39] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:30 PM >2013-03-11 12:30:02,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [430467a3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 261e3907 >2013-03-11 12:30:02,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [430467a3] FINISH, GetHardwareInfoVDSCommand, log id: 261e3907 >2013-03-11 12:30:03,035 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-65) [f4edd96] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 12:30:03,038 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-65) [14fb4d2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 12:30:03,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7caa8209 >2013-03-11 12:30:09,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 12:30:09,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:30:09,086 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 12:30:09,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 7caa8209 >2013-03-11 12:30:09,090 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [2af52f1c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:30:09,104 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [2af52f1c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 44599b49 >2013-03-11 12:30:09,104 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [2af52f1c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 12:30:09,104 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [2af52f1c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 44599b49 >2013-03-11 12:30:09,106 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [2af52f1c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 12:30:09,106 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [14fb4d2] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 12:30:09,117 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-65) [4de03c8e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:30:09,118 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [4de03c8e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2c6d810b >2013-03-11 12:30:09,119 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [4de03c8e] FINISH, SetVdsStatusVDSCommand, log id: 2c6d810b >2013-03-11 12:30:09,125 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-65) [4de03c8e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 12:30:09,149 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-65) [20c22af4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:30:09,151 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-65) [1fd27d46] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:30:09,153 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-65) [1fd27d46] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 12:30:33,886 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-44) [13b5f2cd] Start refreshing all users data >2013-03-11 12:30:35,080 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [6354c15d] Attempting to update VMs/Templates Ovf. >2013-03-11 12:30:35,081 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [6354c15d] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 12:30:35,088 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [6354c15d] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 12:30:35,088 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [6354c15d] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 12:30:35,091 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [6354c15d] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 12:30:35,091 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [6354c15d] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 12:30:35,092 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-62) [6354c15d] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 12:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) [1b3a9f02] Autorecovering 1 hosts >2013-03-11 12:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) [1b3a9f02] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-97) [61828f5a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-97) [61828f5a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [61828f5a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6e868ae1 >2013-03-11 12:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [61828f5a] FINISH, SetVdsStatusVDSCommand, log id: 6e868ae1 >2013-03-11 12:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-97) [61828f5a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 62df7b4e >2013-03-11 12:35:00,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [61828f5a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1f248651 >2013-03-11 12:35:00,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [61828f5a] FINISH, GetHardwareInfoVDSCommand, log id: 1f248651 >2013-03-11 12:35:00,478 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-97) [7d054e0e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:35:00,480 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-97) [3dacdf3a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:35:00,482 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-97) [3dacdf3a] FINISH, ActivateVdsVDSCommand, log id: 62df7b4e >2013-03-11 12:35:00,501 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-97) [3dacdf3a] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:35:00,502 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-97) [3dacdf3a] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 12:35:00,503 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) [3dacdf3a] Autorecovering 1 storage domains >2013-03-11 12:35:00,503 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-97) [3dacdf3a] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 12:35:00,504 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-97) [407f3f2b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:35:00,504 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-97) [407f3f2b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:35 PM >2013-03-11 12:35:00,507 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-97) [407f3f2b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:35 PM >2013-03-11 12:35:00,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [d80ce96] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 51807454 >2013-03-11 12:35:00,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [d80ce96] FINISH, GetHardwareInfoVDSCommand, log id: 51807454 >2013-03-11 12:35:00,579 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-25) [71929da3] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 12:35:00,583 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-25) [17f87977] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 12:35:00,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3692d6a8 >2013-03-11 12:35:05,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 12:35:05,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:35:05,553 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 12:35:05,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 3692d6a8 >2013-03-11 12:35:05,557 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [5e71af99] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:35:05,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [5e71af99] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 730b598a >2013-03-11 12:35:05,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [5e71af99] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 12:35:05,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [5e71af99] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 730b598a >2013-03-11 12:35:05,573 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [5e71af99] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 12:35:05,573 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [17f87977] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 12:35:05,582 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-25) [66d4de00] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:35:05,584 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [66d4de00] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6b374d8b >2013-03-11 12:35:05,585 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [66d4de00] FINISH, SetVdsStatusVDSCommand, log id: 6b374d8b >2013-03-11 12:35:05,590 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-25) [66d4de00] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 12:35:05,615 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-25) [4521fb11] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:35:05,618 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-25) [1d98e869] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:35:05,619 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-25) [1d98e869] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 12:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [4d79dedf] Autorecovering 1 hosts >2013-03-11 12:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [4d79dedf] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [47d5e758] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [47d5e758] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:40:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [47d5e758] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4f5259f7 >2013-03-11 12:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [47d5e758] FINISH, SetVdsStatusVDSCommand, log id: 4f5259f7 >2013-03-11 12:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-46) [47d5e758] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: afa4bbe >2013-03-11 12:40:00,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [47d5e758] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 311fa74 >2013-03-11 12:40:00,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [47d5e758] FINISH, GetHardwareInfoVDSCommand, log id: 311fa74 >2013-03-11 12:40:00,090 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-46) [7b453696] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:40:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-46) [48563a42] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:40:00,094 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-46) [48563a42] FINISH, ActivateVdsVDSCommand, log id: afa4bbe >2013-03-11 12:40:00,113 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [48563a42] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:40:00,114 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-46) [48563a42] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 12:40:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [48563a42] Autorecovering 1 storage domains >2013-03-11 12:40:00,115 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [48563a42] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 12:40:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [6a863ebd] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:40:00,116 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [6a863ebd] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:40 PM >2013-03-11 12:40:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [6a863ebd] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:40 PM >2013-03-11 12:40:02,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-3) [a488127] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 25d3b6b9 >2013-03-11 12:40:02,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-3) [a488127] FINISH, GetHardwareInfoVDSCommand, log id: 25d3b6b9 >2013-03-11 12:40:02,907 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-3) [156bad29] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 12:40:02,911 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-3) [fc3842c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 12:40:02,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 52be9761 >2013-03-11 12:40:09,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 12:40:09,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:40:09,938 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 12:40:09,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 52be9761 >2013-03-11 12:40:09,942 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [20877e98] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:40:09,956 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [20877e98] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4e2ce528 >2013-03-11 12:40:09,956 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [20877e98] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 12:40:09,956 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [20877e98] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4e2ce528 >2013-03-11 12:40:09,958 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [20877e98] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 12:40:09,959 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [fc3842c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 12:40:09,968 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-3) [761ffbb4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:40:09,969 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-3) [761ffbb4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1533d51e >2013-03-11 12:40:09,971 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-3) [761ffbb4] FINISH, SetVdsStatusVDSCommand, log id: 1533d51e >2013-03-11 12:40:09,976 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) [761ffbb4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 12:40:09,999 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-3) [605baf31] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:40:10,002 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-3) [3c5897be] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:40:10,003 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-3) [3c5897be] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 12:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [574be938] Autorecovering 1 hosts >2013-03-11 12:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [574be938] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [5d5a67ab] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [5d5a67ab] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [5d5a67ab] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 792a1a85 >2013-03-11 12:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [5d5a67ab] FINISH, SetVdsStatusVDSCommand, log id: 792a1a85 >2013-03-11 12:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-57) [5d5a67ab] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 720f3986 >2013-03-11 12:45:00,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [5d5a67ab] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1b718ef8 >2013-03-11 12:45:00,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [5d5a67ab] FINISH, GetHardwareInfoVDSCommand, log id: 1b718ef8 >2013-03-11 12:45:00,257 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-57) [bede303] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:45:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-57) [3783fcd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:45:00,261 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-57) [3783fcd] FINISH, ActivateVdsVDSCommand, log id: 720f3986 >2013-03-11 12:45:00,280 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [3783fcd] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:45:00,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-57) [3783fcd] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 12:45:00,282 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [3783fcd] Autorecovering 1 storage domains >2013-03-11 12:45:00,282 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [3783fcd] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 12:45:00,283 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [4e1c2020] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:45:00,283 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [4e1c2020] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:45 PM >2013-03-11 12:45:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [4e1c2020] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:45 PM >2013-03-11 12:45:01,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [1dc51316] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 624a69ef >2013-03-11 12:45:01,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [1dc51316] FINISH, GetHardwareInfoVDSCommand, log id: 624a69ef >2013-03-11 12:45:01,284 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-43) [55d2369b] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 12:45:01,287 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-43) [260c21f8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 12:45:01,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4f1d0b77 >2013-03-11 12:45:06,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 12:45:06,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:45:06,284 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 12:45:06,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 4f1d0b77 >2013-03-11 12:45:06,288 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [3c0b7f9b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:45:06,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [3c0b7f9b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6d0623c2 >2013-03-11 12:45:06,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [3c0b7f9b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 12:45:06,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [3c0b7f9b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6d0623c2 >2013-03-11 12:45:06,304 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [3c0b7f9b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 12:45:06,304 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [260c21f8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 12:45:06,314 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-43) [307c9a70] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:45:06,315 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [307c9a70] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2609502 >2013-03-11 12:45:06,316 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [307c9a70] FINISH, SetVdsStatusVDSCommand, log id: 2609502 >2013-03-11 12:45:06,322 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [307c9a70] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 12:45:06,345 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [f1b2914] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:45:06,348 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [2f602012] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:45:06,349 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-43) [2f602012] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 12:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [44b36dbb] Autorecovering 1 hosts >2013-03-11 12:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [44b36dbb] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [327b4681] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [327b4681] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [327b4681] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1630dac2 >2013-03-11 12:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [327b4681] FINISH, SetVdsStatusVDSCommand, log id: 1630dac2 >2013-03-11 12:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [327b4681] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 25a78023 >2013-03-11 12:50:00,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [327b4681] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 413988ae >2013-03-11 12:50:00,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [327b4681] FINISH, GetHardwareInfoVDSCommand, log id: 413988ae >2013-03-11 12:50:00,183 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [b2d0150] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:50:00,185 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [47bf06c7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:50:00,187 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-72) [47bf06c7] FINISH, ActivateVdsVDSCommand, log id: 25a78023 >2013-03-11 12:50:00,206 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-72) [47bf06c7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:50:00,207 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [47bf06c7] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 12:50:00,208 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [47bf06c7] Autorecovering 1 storage domains >2013-03-11 12:50:00,208 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-72) [47bf06c7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 12:50:00,209 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [395a5ae0] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:50:00,209 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [395a5ae0] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:50 PM >2013-03-11 12:50:00,212 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-72) [395a5ae0] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:50 PM >2013-03-11 12:50:00,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-23) [1b1b0943] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2851390e >2013-03-11 12:50:00,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-23) [1b1b0943] FINISH, GetHardwareInfoVDSCommand, log id: 2851390e >2013-03-11 12:50:00,630 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-23) [5c3be499] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 12:50:00,634 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-23) [884186e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 12:50:00,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5fbf60e7 >2013-03-11 12:50:05,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 12:50:05,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:50:05,641 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 12:50:05,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 5fbf60e7 >2013-03-11 12:50:05,645 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [2c62ecf0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:50:05,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [2c62ecf0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40b47503 >2013-03-11 12:50:05,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [2c62ecf0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 12:50:05,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [2c62ecf0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40b47503 >2013-03-11 12:50:05,661 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [2c62ecf0] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 12:50:05,661 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [884186e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 12:50:05,670 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-23) [55dd4b09] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:50:05,672 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-23) [55dd4b09] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2c96c0e0 >2013-03-11 12:50:05,673 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-23) [55dd4b09] FINISH, SetVdsStatusVDSCommand, log id: 2c96c0e0 >2013-03-11 12:50:05,679 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-23) [55dd4b09] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 12:50:05,703 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-23) [16d65cb4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:50:05,705 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-23) [479b339f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:50:05,706 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-23) [479b339f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 12:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [50a5b579] Autorecovering 1 hosts >2013-03-11 12:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [50a5b579] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [36c7f254] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [36c7f254] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [36c7f254] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1a2e0160 >2013-03-11 12:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [36c7f254] FINISH, SetVdsStatusVDSCommand, log id: 1a2e0160 >2013-03-11 12:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-83) [36c7f254] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6ccc0f54 >2013-03-11 12:55:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [36c7f254] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3af82976 >2013-03-11 12:55:00,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [36c7f254] FINISH, GetHardwareInfoVDSCommand, log id: 3af82976 >2013-03-11 12:55:00,219 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-83) [18175c60] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:55:00,222 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-83) [58f618cb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:55:00,223 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-83) [58f618cb] FINISH, ActivateVdsVDSCommand, log id: 6ccc0f54 >2013-03-11 12:55:00,242 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-83) [58f618cb] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 12:55:00,243 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-83) [58f618cb] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 12:55:00,244 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [58f618cb] Autorecovering 1 storage domains >2013-03-11 12:55:00,245 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-83) [58f618cb] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 12:55:00,245 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [7176d9d0] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:55:00,245 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [7176d9d0] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 12:55 PM >2013-03-11 12:55:00,248 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-83) [7176d9d0] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 12:55 PM >2013-03-11 12:55:02,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [79cd6a09] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 65d08f6e >2013-03-11 12:55:02,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [79cd6a09] FINISH, GetHardwareInfoVDSCommand, log id: 65d08f6e >2013-03-11 12:55:03,021 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-75) [56ead29e] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 12:55:03,025 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-75) [265e770c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 12:55:03,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 448b3bc3 >2013-03-11 12:55:09,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 12:55:09,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 12:55:09,027 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 12:55:09,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 448b3bc3 >2013-03-11 12:55:09,030 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [11da2de0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 12:55:09,052 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [11da2de0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2c070d3c >2013-03-11 12:55:09,053 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [11da2de0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 12:55:09,053 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [11da2de0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2c070d3c >2013-03-11 12:55:09,057 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [265e770c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 12:55:09,067 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-75) [2f4fd58f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:55:09,069 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [2f4fd58f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3175b534 >2013-03-11 12:55:09,070 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [2f4fd58f] FINISH, SetVdsStatusVDSCommand, log id: 3175b534 >2013-03-11 12:55:09,075 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-75) [2f4fd58f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 12:55:09,098 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [3b0384cb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:55:09,100 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [3652e5ee] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 12:55:09,102 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-75) [3652e5ee] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 13:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [9046d64] Autorecovering 1 hosts >2013-03-11 13:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [9046d64] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [30c9da60] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [30c9da60] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [30c9da60] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 223ac95c >2013-03-11 13:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [30c9da60] FINISH, SetVdsStatusVDSCommand, log id: 223ac95c >2013-03-11 13:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-9) [30c9da60] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 45e42d9 >2013-03-11 13:00:00,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [30c9da60] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7624e129 >2013-03-11 13:00:00,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [30c9da60] FINISH, GetHardwareInfoVDSCommand, log id: 7624e129 >2013-03-11 13:00:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-9) [6bd52cef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:00:00,263 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-9) [764846c9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:00:00,264 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-9) [764846c9] FINISH, ActivateVdsVDSCommand, log id: 45e42d9 >2013-03-11 13:00:00,286 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [764846c9] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:00:00,287 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-9) [764846c9] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 13:00:00,288 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [764846c9] Autorecovering 1 storage domains >2013-03-11 13:00:00,288 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [764846c9] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 13:00:00,289 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [1f79cd8c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:00:00,289 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [1f79cd8c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:00 PM >2013-03-11 13:00:00,291 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [1f79cd8c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:00 PM >2013-03-11 13:00:00,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [e877d39] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 53baf94d >2013-03-11 13:00:00,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [e877d39] FINISH, GetHardwareInfoVDSCommand, log id: 53baf94d >2013-03-11 13:00:00,375 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-79) [6efad9c3] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 13:00:00,379 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-79) [45d42681] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 13:00:00,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 158eeb3d >2013-03-11 13:00:07,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 13:00:07,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:00:07,407 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 13:00:07,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 158eeb3d >2013-03-11 13:00:07,411 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [264b7c30] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:00:07,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [264b7c30] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 77ece20 >2013-03-11 13:00:07,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [264b7c30] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 13:00:07,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [264b7c30] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 77ece20 >2013-03-11 13:00:07,427 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [264b7c30] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 13:00:07,427 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [45d42681] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 13:00:07,437 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-79) [460c6f17] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:00:07,438 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [460c6f17] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7a855a10 >2013-03-11 13:00:07,440 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [460c6f17] FINISH, SetVdsStatusVDSCommand, log id: 7a855a10 >2013-03-11 13:00:07,445 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-79) [460c6f17] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 13:00:07,468 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-79) [6d1e03d4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:00:07,470 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-79) [7cb642fe] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:00:07,472 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-79) [7cb642fe] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 13:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [704d8ace] Autorecovering 1 hosts >2013-03-11 13:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [704d8ace] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [6da65392] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [6da65392] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [6da65392] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5c4536cd >2013-03-11 13:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [6da65392] FINISH, SetVdsStatusVDSCommand, log id: 5c4536cd >2013-03-11 13:05:00,010 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-88) [6da65392] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3001673d >2013-03-11 13:05:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [6da65392] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 57af20ea >2013-03-11 13:05:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [6da65392] FINISH, GetHardwareInfoVDSCommand, log id: 57af20ea >2013-03-11 13:05:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-88) [740ef73] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:05:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-88) [5262bc5c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:05:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-88) [5262bc5c] FINISH, ActivateVdsVDSCommand, log id: 3001673d >2013-03-11 13:05:00,221 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [5262bc5c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:05:00,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-88) [5262bc5c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 13:05:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [5262bc5c] Autorecovering 1 storage domains >2013-03-11 13:05:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [5262bc5c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 13:05:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [1eec12ea] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:05:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [1eec12ea] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:05 PM >2013-03-11 13:05:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [1eec12ea] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:05 PM >2013-03-11 13:05:01,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [395a5ae0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 46a77353 >2013-03-11 13:05:01,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [395a5ae0] FINISH, GetHardwareInfoVDSCommand, log id: 46a77353 >2013-03-11 13:05:01,749 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-72) [14b08dd4] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 13:05:01,752 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-72) [7f495e98] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 13:05:01,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 71e086c7 >2013-03-11 13:05:08,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 13:05:08,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:05:08,764 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 13:05:08,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 71e086c7 >2013-03-11 13:05:08,767 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [135cef7a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:05:08,781 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [135cef7a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 611edef8 >2013-03-11 13:05:08,781 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [135cef7a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 13:05:08,781 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [135cef7a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 611edef8 >2013-03-11 13:05:08,783 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [135cef7a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 13:05:08,784 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [7f495e98] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 13:05:08,793 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-72) [3980fce0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:05:08,794 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [3980fce0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 60bbdd4d >2013-03-11 13:05:08,796 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [3980fce0] FINISH, SetVdsStatusVDSCommand, log id: 60bbdd4d >2013-03-11 13:05:08,802 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [3980fce0] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 13:05:08,826 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [486bc7ae] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:05:08,829 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [547d49ab] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:05:08,830 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-72) [547d49ab] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 13:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [2121b38e] Autorecovering 1 hosts >2013-03-11 13:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [2121b38e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-60) [30451958] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-60) [30451958] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [30451958] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: ddb724c >2013-03-11 13:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [30451958] FINISH, SetVdsStatusVDSCommand, log id: ddb724c >2013-03-11 13:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-60) [30451958] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3cdca64b >2013-03-11 13:10:00,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [30451958] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3c8234b0 >2013-03-11 13:10:00,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [30451958] FINISH, GetHardwareInfoVDSCommand, log id: 3c8234b0 >2013-03-11 13:10:00,259 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-60) [64f6e49f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:10:00,261 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-60) [7165abc8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:10:00,263 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-60) [7165abc8] FINISH, ActivateVdsVDSCommand, log id: 3cdca64b >2013-03-11 13:10:00,282 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-60) [7165abc8] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:10:00,283 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-60) [7165abc8] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 13:10:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [7165abc8] Autorecovering 1 storage domains >2013-03-11 13:10:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [7165abc8] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 13:10:00,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-60) [29f44e07] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:10:00,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-60) [29f44e07] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:10 PM >2013-03-11 13:10:00,287 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-60) [29f44e07] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:10 PM >2013-03-11 13:10:03,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [50a62fe] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 58be4994 >2013-03-11 13:10:03,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [50a62fe] FINISH, GetHardwareInfoVDSCommand, log id: 58be4994 >2013-03-11 13:10:03,107 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-19) [4b057dc6] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 13:10:03,110 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-19) [668db14d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 13:10:03,120 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4c0ae4d4 >2013-03-11 13:10:09,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 13:10:09,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:10:09,091 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 13:10:09,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 4c0ae4d4 >2013-03-11 13:10:09,095 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [188effe0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:10:09,108 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [188effe0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3750488b >2013-03-11 13:10:09,108 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [188effe0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 13:10:09,108 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [188effe0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3750488b >2013-03-11 13:10:09,110 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [188effe0] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 13:10:09,111 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [668db14d] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 13:10:09,120 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-19) [2c75affd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:10:09,122 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [2c75affd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 79f2b05a >2013-03-11 13:10:09,123 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [2c75affd] FINISH, SetVdsStatusVDSCommand, log id: 79f2b05a >2013-03-11 13:10:09,128 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-19) [2c75affd] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 13:10:09,163 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-19) [ac938a6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:10:09,165 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-19) [d9efe9e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:10:09,167 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-19) [d9efe9e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 13:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [668b4021] Autorecovering 1 hosts >2013-03-11 13:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [668b4021] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [555a1e4a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [555a1e4a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [555a1e4a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1f3ded97 >2013-03-11 13:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-32) [555a1e4a] FINISH, SetVdsStatusVDSCommand, log id: 1f3ded97 >2013-03-11 13:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-32) [555a1e4a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4fafd6ea >2013-03-11 13:15:00,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [555a1e4a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5490a122 >2013-03-11 13:15:00,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-32) [555a1e4a] FINISH, GetHardwareInfoVDSCommand, log id: 5490a122 >2013-03-11 13:15:00,246 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-32) [3d1104f1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:15:00,249 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-32) [401df60e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:15:00,251 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-32) [401df60e] FINISH, ActivateVdsVDSCommand, log id: 4fafd6ea >2013-03-11 13:15:00,276 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-32) [401df60e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:15:00,277 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-32) [401df60e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 13:15:00,281 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [401df60e] Autorecovering 1 storage domains >2013-03-11 13:15:00,281 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-32) [401df60e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 13:15:00,283 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [56ec593d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:15:00,283 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [56ec593d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:15 PM >2013-03-11 13:15:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-32) [56ec593d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:15 PM >2013-03-11 13:15:00,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [6d22a015] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7b6fc86c >2013-03-11 13:15:00,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-41) [6d22a015] FINISH, GetHardwareInfoVDSCommand, log id: 7b6fc86c >2013-03-11 13:15:00,450 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-41) [3c4e4918] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 13:15:00,454 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-41) [2f2effee] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 13:15:00,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 422b5128 >2013-03-11 13:15:06,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 13:15:06,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:15:06,441 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 13:15:06,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 422b5128 >2013-03-11 13:15:06,444 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [6eba4d34] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:15:06,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [6eba4d34] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: ff5f9d >2013-03-11 13:15:06,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [6eba4d34] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 13:15:06,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [6eba4d34] FINISH, MarkPoolInReconstructModeVDSCommand, log id: ff5f9d >2013-03-11 13:15:06,465 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [6eba4d34] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 13:15:06,466 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [2f2effee] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 13:15:06,477 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-41) [4c9e8504] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:15:06,479 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [4c9e8504] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 20f80b29 >2013-03-11 13:15:06,480 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-41) [4c9e8504] FINISH, SetVdsStatusVDSCommand, log id: 20f80b29 >2013-03-11 13:15:06,486 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-41) [4c9e8504] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 13:15:06,511 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-41) [5930f58e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:15:06,513 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-41) [3d4a3bd8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:15:06,514 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-41) [3d4a3bd8] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 13:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [7fb2ac54] Autorecovering 1 hosts >2013-03-11 13:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [7fb2ac54] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [1d402af6] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [1d402af6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [1d402af6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7090679e >2013-03-11 13:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [1d402af6] FINISH, SetVdsStatusVDSCommand, log id: 7090679e >2013-03-11 13:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-27) [1d402af6] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4d75bcf1 >2013-03-11 13:20:00,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [1d402af6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 42288ce3 >2013-03-11 13:20:00,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [1d402af6] FINISH, GetHardwareInfoVDSCommand, log id: 42288ce3 >2013-03-11 13:20:00,245 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-27) [302a58e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:20:00,247 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-27) [259ba294] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:20:00,249 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-27) [259ba294] FINISH, ActivateVdsVDSCommand, log id: 4d75bcf1 >2013-03-11 13:20:00,268 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [259ba294] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:20:00,269 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-27) [259ba294] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 13:20:00,270 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [259ba294] Autorecovering 1 storage domains >2013-03-11 13:20:00,270 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [259ba294] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 13:20:00,271 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [5a5db588] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:20:00,271 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [5a5db588] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:20 PM >2013-03-11 13:20:00,273 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [5a5db588] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:20 PM >2013-03-11 13:20:00,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [4898fbf6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1729275d >2013-03-11 13:20:00,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [4898fbf6] FINISH, GetHardwareInfoVDSCommand, log id: 1729275d >2013-03-11 13:20:00,788 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-63) [5e5c50b] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 13:20:00,792 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-63) [5cdd75c8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 13:20:00,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6c25cb2d >2013-03-11 13:20:07,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 13:20:07,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:20:07,788 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 13:20:07,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 6c25cb2d >2013-03-11 13:20:07,792 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [11acc1c3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:20:07,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [11acc1c3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6dcb4b48 >2013-03-11 13:20:07,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [11acc1c3] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 13:20:07,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [11acc1c3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6dcb4b48 >2013-03-11 13:20:07,808 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [11acc1c3] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 13:20:07,809 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [5cdd75c8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 13:20:07,818 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-63) [7b5ec960] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:20:07,819 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [7b5ec960] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1dae0e0 >2013-03-11 13:20:07,821 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [7b5ec960] FINISH, SetVdsStatusVDSCommand, log id: 1dae0e0 >2013-03-11 13:20:07,827 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-63) [7b5ec960] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 13:20:07,878 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-63) [74022f19] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:20:07,881 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-63) [2431912f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:20:07,882 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-63) [2431912f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 13:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [1bfd15b9] Autorecovering 1 hosts >2013-03-11 13:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [1bfd15b9] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [50102fda] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [50102fda] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:25:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [50102fda] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 19d2af7f >2013-03-11 13:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-96) [50102fda] FINISH, SetVdsStatusVDSCommand, log id: 19d2af7f >2013-03-11 13:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [50102fda] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4083a3a9 >2013-03-11 13:25:00,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [50102fda] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 51b1bec3 >2013-03-11 13:25:00,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-96) [50102fda] FINISH, GetHardwareInfoVDSCommand, log id: 51b1bec3 >2013-03-11 13:25:00,264 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-96) [51e74030] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:25:00,267 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-96) [7bf78a5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:25:00,268 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-96) [7bf78a5] FINISH, ActivateVdsVDSCommand, log id: 4083a3a9 >2013-03-11 13:25:00,288 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-96) [7bf78a5] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:25:00,288 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-96) [7bf78a5] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 13:25:00,290 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [7bf78a5] Autorecovering 1 storage domains >2013-03-11 13:25:00,290 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-96) [7bf78a5] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 13:25:00,290 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [4e36b1a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:25:00,290 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [4e36b1a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:25 PM >2013-03-11 13:25:00,293 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-96) [4e36b1a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:25 PM >2013-03-11 13:25:02,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [1e052ef9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6bef63b >2013-03-11 13:25:02,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-12) [1e052ef9] FINISH, GetHardwareInfoVDSCommand, log id: 6bef63b >2013-03-11 13:25:02,166 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-12) [30a756d4] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 13:25:02,170 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-12) [7562f6d9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 13:25:02,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 29e8e260 >2013-03-11 13:25:09,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 13:25:09,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:25:09,164 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 13:25:09,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 29e8e260 >2013-03-11 13:25:09,168 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [1a31dc9c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:25:09,181 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [1a31dc9c] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: cbc07c0 >2013-03-11 13:25:09,181 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [1a31dc9c] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 13:25:09,181 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [1a31dc9c] FINISH, MarkPoolInReconstructModeVDSCommand, log id: cbc07c0 >2013-03-11 13:25:09,183 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [1a31dc9c] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 13:25:09,184 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [7562f6d9] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 13:25:09,193 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-12) [694f3cf2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:25:09,195 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [694f3cf2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 304daf5f >2013-03-11 13:25:09,196 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-12) [694f3cf2] FINISH, SetVdsStatusVDSCommand, log id: 304daf5f >2013-03-11 13:25:09,202 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-12) [694f3cf2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 13:25:09,225 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-12) [6f9def5e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:25:09,227 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-12) [66a03382] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:25:09,229 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-12) [66a03382] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 13:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [37a992da] Autorecovering 1 hosts >2013-03-11 13:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [37a992da] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [758e91f7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [758e91f7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [758e91f7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1f906f6f >2013-03-11 13:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [758e91f7] FINISH, SetVdsStatusVDSCommand, log id: 1f906f6f >2013-03-11 13:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-77) [758e91f7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 229af611 >2013-03-11 13:30:00,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [758e91f7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 31cb49e3 >2013-03-11 13:30:00,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [758e91f7] FINISH, GetHardwareInfoVDSCommand, log id: 31cb49e3 >2013-03-11 13:30:00,267 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-77) [5c33ae76] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:30:00,269 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-77) [74259ca2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:30:00,270 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-77) [74259ca2] FINISH, ActivateVdsVDSCommand, log id: 229af611 >2013-03-11 13:30:00,290 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [74259ca2] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:30:00,291 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-77) [74259ca2] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 13:30:00,292 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [74259ca2] Autorecovering 1 storage domains >2013-03-11 13:30:00,292 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [74259ca2] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 13:30:00,293 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [11bfc989] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:30:00,293 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [11bfc989] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:30 PM >2013-03-11 13:30:00,296 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [11bfc989] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:30 PM >2013-03-11 13:30:00,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [69312670] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 581dbc24 >2013-03-11 13:30:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [69312670] FINISH, GetHardwareInfoVDSCommand, log id: 581dbc24 >2013-03-11 13:30:00,512 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-22) [1b8bfa9e] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 13:30:00,515 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-22) [1c5b8133] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 13:30:00,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4d51b55 >2013-03-11 13:30:08,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 13:30:08,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:30:08,561 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 13:30:08,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 4d51b55 >2013-03-11 13:30:08,565 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [65ac5416] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:30:08,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [65ac5416] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4d3edb5f >2013-03-11 13:30:08,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [65ac5416] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 13:30:08,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [65ac5416] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4d3edb5f >2013-03-11 13:30:08,582 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [65ac5416] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 13:30:08,582 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [1c5b8133] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 13:30:08,592 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-22) [1e8dff34] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:30:08,593 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [1e8dff34] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 365357a3 >2013-03-11 13:30:08,595 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [1e8dff34] FINISH, SetVdsStatusVDSCommand, log id: 365357a3 >2013-03-11 13:30:08,601 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [1e8dff34] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 13:30:08,625 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [765f1153] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:30:08,628 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [18412d77] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:30:08,629 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-22) [18412d77] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 13:30:33,888 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-60) [29f44e07] Start refreshing all users data >2013-03-11 13:30:35,092 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-44) [13b5f2cd] Attempting to update VMs/Templates Ovf. >2013-03-11 13:30:35,093 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-44) [13b5f2cd] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 13:30:35,094 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-44) [13b5f2cd] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 13:30:35,094 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-44) [13b5f2cd] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 13:30:35,095 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-44) [13b5f2cd] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 13:30:35,095 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-44) [13b5f2cd] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 13:30:35,096 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-44) [13b5f2cd] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 13:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [5a5db588] Autorecovering 1 hosts >2013-03-11 13:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [5a5db588] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [1aa84a54] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [1aa84a54] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [1aa84a54] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3420849f >2013-03-11 13:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-27) [1aa84a54] FINISH, SetVdsStatusVDSCommand, log id: 3420849f >2013-03-11 13:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-27) [1aa84a54] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5d72891b >2013-03-11 13:35:00,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [1aa84a54] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1f47fc05 >2013-03-11 13:35:00,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-27) [1aa84a54] FINISH, GetHardwareInfoVDSCommand, log id: 1f47fc05 >2013-03-11 13:35:00,290 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-27) [78fb126] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:35:00,293 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-27) [4958a1a9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:35:00,294 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-27) [4958a1a9] FINISH, ActivateVdsVDSCommand, log id: 5d72891b >2013-03-11 13:35:00,313 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-27) [4958a1a9] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:35:00,314 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-27) [4958a1a9] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 13:35:00,315 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [4958a1a9] Autorecovering 1 storage domains >2013-03-11 13:35:00,315 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-27) [4958a1a9] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 13:35:00,316 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [3b7b20e9] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:35:00,316 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [3b7b20e9] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:35 PM >2013-03-11 13:35:00,319 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-27) [3b7b20e9] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:35 PM >2013-03-11 13:35:02,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [79d8b398] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 36f79015 >2013-03-11 13:35:02,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [79d8b398] FINISH, GetHardwareInfoVDSCommand, log id: 36f79015 >2013-03-11 13:35:02,914 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-61) [3acca172] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 13:35:02,918 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-61) [6500630b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 13:35:02,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 76a37548 >2013-03-11 13:35:10,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 13:35:10,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:35:10,901 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 13:35:10,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 76a37548 >2013-03-11 13:35:10,905 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [7de3d8e2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:35:10,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [7de3d8e2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 62189822 >2013-03-11 13:35:10,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [7de3d8e2] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 13:35:10,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [7de3d8e2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 62189822 >2013-03-11 13:35:10,921 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [7de3d8e2] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 13:35:10,921 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-45) [6500630b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 13:35:10,930 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-61) [6ab1a4d0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:35:10,932 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [6ab1a4d0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: c7d763d >2013-03-11 13:35:10,933 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [6ab1a4d0] FINISH, SetVdsStatusVDSCommand, log id: c7d763d >2013-03-11 13:35:10,939 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-61) [6ab1a4d0] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 13:35:10,965 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-61) [2f993182] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:35:10,967 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-61) [617028df] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:35:10,968 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-61) [617028df] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 13:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [4f44faa2] Autorecovering 1 hosts >2013-03-11 13:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [4f44faa2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [f1a8645] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [f1a8645] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [f1a8645] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 76e5fa59 >2013-03-11 13:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-17) [f1a8645] FINISH, SetVdsStatusVDSCommand, log id: 76e5fa59 >2013-03-11 13:40:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [f1a8645] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 65d18e65 >2013-03-11 13:40:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [f1a8645] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1a4a8892 >2013-03-11 13:40:00,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-17) [f1a8645] FINISH, GetHardwareInfoVDSCommand, log id: 1a4a8892 >2013-03-11 13:40:00,259 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-17) [70053b6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:40:00,261 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-17) [3186b6d7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:40:00,262 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-17) [3186b6d7] FINISH, ActivateVdsVDSCommand, log id: 65d18e65 >2013-03-11 13:40:00,281 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-17) [3186b6d7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:40:00,282 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-17) [3186b6d7] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 13:40:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [3186b6d7] Autorecovering 1 storage domains >2013-03-11 13:40:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-17) [3186b6d7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 13:40:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [7eb8a541] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:40:00,284 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [7eb8a541] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:40 PM >2013-03-11 13:40:00,287 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-17) [7eb8a541] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:40 PM >2013-03-11 13:40:02,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [5f47c13d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 11caa8c0 >2013-03-11 13:40:02,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [5f47c13d] FINISH, GetHardwareInfoVDSCommand, log id: 11caa8c0 >2013-03-11 13:40:02,249 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-95) [34f91272] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 13:40:02,252 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-95) [3d086d9e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 13:40:02,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2765a5f9 >2013-03-11 13:40:09,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 13:40:09,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:40:09,237 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 13:40:09,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 2765a5f9 >2013-03-11 13:40:09,241 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [f505bb2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:40:09,254 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [f505bb2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: d5ba901 >2013-03-11 13:40:09,254 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [f505bb2] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 13:40:09,254 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [f505bb2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: d5ba901 >2013-03-11 13:40:09,256 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [f505bb2] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 13:40:09,257 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [3d086d9e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 13:40:09,266 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-95) [59b7b4f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:40:09,267 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [59b7b4f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3c363ea0 >2013-03-11 13:40:09,268 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [59b7b4f] FINISH, SetVdsStatusVDSCommand, log id: 3c363ea0 >2013-03-11 13:40:09,274 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-95) [59b7b4f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 13:40:09,297 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-95) [36b23d26] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:40:09,299 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-95) [7cf91ca0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:40:09,300 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-95) [7cf91ca0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 13:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [6dbba9f1] Autorecovering 1 hosts >2013-03-11 13:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [6dbba9f1] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-76) [7cb87fdc] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-76) [7cb87fdc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [7cb87fdc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2534111d >2013-03-11 13:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-76) [7cb87fdc] FINISH, SetVdsStatusVDSCommand, log id: 2534111d >2013-03-11 13:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-76) [7cb87fdc] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 600be074 >2013-03-11 13:45:00,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [7cb87fdc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7693ac1a >2013-03-11 13:45:00,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-76) [7cb87fdc] FINISH, GetHardwareInfoVDSCommand, log id: 7693ac1a >2013-03-11 13:45:00,187 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-76) [5ea9c8a4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:45:00,190 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-76) [4aa854f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:45:00,191 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-76) [4aa854f] FINISH, ActivateVdsVDSCommand, log id: 600be074 >2013-03-11 13:45:00,210 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-76) [4aa854f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:45:00,211 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-76) [4aa854f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 13:45:00,212 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [4aa854f] Autorecovering 1 storage domains >2013-03-11 13:45:00,212 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-76) [4aa854f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 13:45:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-76) [25a7e614] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:45:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-76) [25a7e614] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:45 PM >2013-03-11 13:45:00,216 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-76) [25a7e614] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:45 PM >2013-03-11 13:45:00,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [695ed367] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 38f02aa2 >2013-03-11 13:45:00,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [695ed367] FINISH, GetHardwareInfoVDSCommand, log id: 38f02aa2 >2013-03-11 13:45:00,579 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-53) [7c04365a] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 13:45:00,582 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-53) [23b09576] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 13:45:00,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4e571b7a >2013-03-11 13:45:05,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 13:45:05,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:45:05,568 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 13:45:05,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 4e571b7a >2013-03-11 13:45:05,572 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [362a1897] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:45:05,586 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [362a1897] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4a3129c9 >2013-03-11 13:45:05,586 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [362a1897] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 13:45:05,586 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [362a1897] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4a3129c9 >2013-03-11 13:45:05,588 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [362a1897] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 13:45:05,588 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [23b09576] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 13:45:05,598 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-53) [5bde7465] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:45:05,599 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [5bde7465] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5a31304b >2013-03-11 13:45:05,601 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [5bde7465] FINISH, SetVdsStatusVDSCommand, log id: 5a31304b >2013-03-11 13:45:05,607 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-53) [5bde7465] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 13:45:05,631 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-53) [142d286d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:45:05,633 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-53) [3d2e9d6d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:45:05,634 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-53) [3d2e9d6d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 13:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [5590818f] Autorecovering 1 hosts >2013-03-11 13:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [5590818f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-55) [404267af] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-55) [404267af] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [404267af] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3c9a790e >2013-03-11 13:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [404267af] FINISH, SetVdsStatusVDSCommand, log id: 3c9a790e >2013-03-11 13:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-55) [404267af] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 392999d1 >2013-03-11 13:50:00,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [404267af] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5c43470f >2013-03-11 13:50:00,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [404267af] FINISH, GetHardwareInfoVDSCommand, log id: 5c43470f >2013-03-11 13:50:00,258 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-55) [78de5fe2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:50:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-55) [7a465436] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:50:00,261 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-55) [7a465436] FINISH, ActivateVdsVDSCommand, log id: 392999d1 >2013-03-11 13:50:00,280 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-55) [7a465436] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:50:00,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-55) [7a465436] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 13:50:00,282 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [7a465436] Autorecovering 1 storage domains >2013-03-11 13:50:00,283 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-55) [7a465436] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 13:50:00,283 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-55) [31822876] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:50:00,283 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-55) [31822876] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:50 PM >2013-03-11 13:50:00,286 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-55) [31822876] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:50 PM >2013-03-11 13:50:02,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [1f24fadb] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5d2122bf >2013-03-11 13:50:02,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [1f24fadb] FINISH, GetHardwareInfoVDSCommand, log id: 5d2122bf >2013-03-11 13:50:02,919 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-64) [397cfc5e] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 13:50:02,922 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-64) [7a6b5bdb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 13:50:02,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 501c950d >2013-03-11 13:50:08,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 13:50:08,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:50:08,891 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 13:50:08,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 501c950d >2013-03-11 13:50:08,895 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [6b980e5d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:50:08,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6b980e5d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 34bf12b >2013-03-11 13:50:08,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [6b980e5d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 13:50:08,908 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6b980e5d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 34bf12b >2013-03-11 13:50:08,910 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [6b980e5d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 13:50:08,911 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [7a6b5bdb] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 13:50:08,920 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-64) [1d0e2936] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:50:08,922 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [1d0e2936] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 50ef7cc7 >2013-03-11 13:50:08,923 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [1d0e2936] FINISH, SetVdsStatusVDSCommand, log id: 50ef7cc7 >2013-03-11 13:50:08,929 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-64) [1d0e2936] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 13:50:08,953 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-64) [34280adf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:50:08,955 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-64) [3ea48131] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:50:08,957 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-64) [3ea48131] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 13:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [23db8bdb] Autorecovering 1 hosts >2013-03-11 13:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [23db8bdb] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [23914ec7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [23914ec7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [23914ec7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2bcf87b5 >2013-03-11 13:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [23914ec7] FINISH, SetVdsStatusVDSCommand, log id: 2bcf87b5 >2013-03-11 13:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [23914ec7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5edc6486 >2013-03-11 13:55:00,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [23914ec7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 361b0a9a >2013-03-11 13:55:00,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [23914ec7] FINISH, GetHardwareInfoVDSCommand, log id: 361b0a9a >2013-03-11 13:55:00,257 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [57a31c85] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:55:00,260 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [2d4a5de5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:55:00,261 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [2d4a5de5] FINISH, ActivateVdsVDSCommand, log id: 5edc6486 >2013-03-11 13:55:00,282 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [2d4a5de5] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 13:55:00,283 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-52) [2d4a5de5] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 13:55:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [2d4a5de5] Autorecovering 1 storage domains >2013-03-11 13:55:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [2d4a5de5] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 13:55:00,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [60f445d8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:55:00,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [60f445d8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 1:55 PM >2013-03-11 13:55:00,287 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [60f445d8] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 1:55 PM >2013-03-11 13:55:00,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [44f88a48] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 37c178c8 >2013-03-11 13:55:00,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [44f88a48] FINISH, GetHardwareInfoVDSCommand, log id: 37c178c8 >2013-03-11 13:55:00,362 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-39) [73ecbb27] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 13:55:00,365 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-39) [768cfc9e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 13:55:00,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 249b0f9f >2013-03-11 13:55:06,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 13:55:06,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 13:55:06,376 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 13:55:06,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 249b0f9f >2013-03-11 13:55:06,380 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [792e5bbb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 13:55:06,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [792e5bbb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2e9d1199 >2013-03-11 13:55:06,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [792e5bbb] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 13:55:06,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [792e5bbb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2e9d1199 >2013-03-11 13:55:06,395 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [792e5bbb] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 13:55:06,396 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-49) [768cfc9e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 13:55:06,406 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-39) [678db354] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:55:06,407 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [678db354] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7cc679bb >2013-03-11 13:55:06,409 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [678db354] FINISH, SetVdsStatusVDSCommand, log id: 7cc679bb >2013-03-11 13:55:06,414 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-39) [678db354] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 13:55:06,439 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-39) [4b0f2f24] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:55:06,441 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-39) [67ed1864] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 13:55:06,443 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-39) [67ed1864] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 14:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [42b35db9] Autorecovering 1 hosts >2013-03-11 14:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [42b35db9] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [3640f7bd] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [3640f7bd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [3640f7bd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 582e4e9d >2013-03-11 14:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [3640f7bd] FINISH, SetVdsStatusVDSCommand, log id: 582e4e9d >2013-03-11 14:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [3640f7bd] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 212a25f0 >2013-03-11 14:00:00,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [3640f7bd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5072a246 >2013-03-11 14:00:00,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [3640f7bd] FINISH, GetHardwareInfoVDSCommand, log id: 5072a246 >2013-03-11 14:00:00,461 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-73) [3facf7e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:00:00,464 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-73) [212547f6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:00:00,465 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-73) [212547f6] FINISH, ActivateVdsVDSCommand, log id: 212a25f0 >2013-03-11 14:00:00,484 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-73) [212547f6] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:00:00,485 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-73) [212547f6] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 14:00:00,486 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [212547f6] Autorecovering 1 storage domains >2013-03-11 14:00:00,486 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-73) [212547f6] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 14:00:00,487 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [64f9c4dd] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:00:00,487 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [64f9c4dd] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:00 PM >2013-03-11 14:00:00,490 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-73) [64f9c4dd] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:00 PM >2013-03-11 14:00:00,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [34c31a45] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 55fa5a55 >2013-03-11 14:00:00,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [34c31a45] FINISH, GetHardwareInfoVDSCommand, log id: 55fa5a55 >2013-03-11 14:00:00,717 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-18) [6df2a9ca] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 14:00:00,721 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-18) [12564e64] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 14:00:00,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 666c8c5f >2013-03-11 14:00:08,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 14:00:08,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:00:08,769 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 14:00:08,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 666c8c5f >2013-03-11 14:00:08,772 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [2da39a34] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:00:08,787 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [2da39a34] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3fd69a40 >2013-03-11 14:00:08,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [2da39a34] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 14:00:08,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [2da39a34] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3fd69a40 >2013-03-11 14:00:08,793 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [12564e64] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 14:00:08,807 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-18) [382b3d49] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:00:08,808 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [382b3d49] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 714ec1e6 >2013-03-11 14:00:08,809 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [382b3d49] FINISH, SetVdsStatusVDSCommand, log id: 714ec1e6 >2013-03-11 14:00:08,815 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [382b3d49] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 14:00:08,838 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [235d0ebd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:00:08,840 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [240e1971] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:00:08,842 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) [240e1971] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 14:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [57edfeae] Autorecovering 1 hosts >2013-03-11 14:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [57edfeae] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [79ffdf7c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [79ffdf7c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [79ffdf7c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6e6d42ab >2013-03-11 14:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-21) [79ffdf7c] FINISH, SetVdsStatusVDSCommand, log id: 6e6d42ab >2013-03-11 14:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-21) [79ffdf7c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 780d75ff >2013-03-11 14:05:00,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [79ffdf7c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 264191ed >2013-03-11 14:05:00,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-21) [79ffdf7c] FINISH, GetHardwareInfoVDSCommand, log id: 264191ed >2013-03-11 14:05:00,202 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-21) [1aceb629] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:05:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-21) [7f825c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:05:00,206 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-21) [7f825c] FINISH, ActivateVdsVDSCommand, log id: 780d75ff >2013-03-11 14:05:00,226 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-21) [7f825c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:05:00,227 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-21) [7f825c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 14:05:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [7f825c] Autorecovering 1 storage domains >2013-03-11 14:05:00,228 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-21) [7f825c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 14:05:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [474f0921] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:05:00,229 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [474f0921] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:05 PM >2013-03-11 14:05:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-21) [474f0921] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:05 PM >2013-03-11 14:05:00,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [7b36458a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 692352d1 >2013-03-11 14:05:00,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [7b36458a] FINISH, GetHardwareInfoVDSCommand, log id: 692352d1 >2013-03-11 14:05:00,306 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-87) [69480a21] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 14:05:00,309 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-87) [356fe346] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 14:05:00,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 782961dc >2013-03-11 14:05:07,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 14:05:07,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:05:07,663 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 14:05:07,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 782961dc >2013-03-11 14:05:07,667 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [507f73d0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:05:07,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [507f73d0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 23c6e501 >2013-03-11 14:05:07,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [507f73d0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 14:05:07,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [507f73d0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 23c6e501 >2013-03-11 14:05:07,683 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [507f73d0] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 14:05:07,683 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [356fe346] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 14:05:07,692 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-87) [5f1d3223] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:05:07,694 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [5f1d3223] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 714d31d8 >2013-03-11 14:05:07,695 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [5f1d3223] FINISH, SetVdsStatusVDSCommand, log id: 714d31d8 >2013-03-11 14:05:07,701 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [5f1d3223] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 14:05:07,724 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [4b36ad4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:05:07,727 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [63f7a017] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:05:07,728 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-87) [63f7a017] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 14:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [33d29aab] Autorecovering 1 hosts >2013-03-11 14:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [33d29aab] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [5d23ea76] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [5d23ea76] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [5d23ea76] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4e24d586 >2013-03-11 14:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [5d23ea76] FINISH, SetVdsStatusVDSCommand, log id: 4e24d586 >2013-03-11 14:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [5d23ea76] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4dc992ab >2013-03-11 14:10:00,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [5d23ea76] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 43698f91 >2013-03-11 14:10:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [5d23ea76] FINISH, GetHardwareInfoVDSCommand, log id: 43698f91 >2013-03-11 14:10:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-14) [33d14bd3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:10:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-14) [62bffdc9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:10:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-14) [62bffdc9] FINISH, ActivateVdsVDSCommand, log id: 4dc992ab >2013-03-11 14:10:00,221 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-14) [62bffdc9] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:10:00,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-14) [62bffdc9] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 14:10:00,223 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [62bffdc9] Autorecovering 1 storage domains >2013-03-11 14:10:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-14) [62bffdc9] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 14:10:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [15116a36] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:10:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [15116a36] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:10 PM >2013-03-11 14:10:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-14) [15116a36] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:10 PM >2013-03-11 14:10:01,961 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [40ec6559] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 77c9785e >2013-03-11 14:10:01,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-37) [40ec6559] FINISH, GetHardwareInfoVDSCommand, log id: 77c9785e >2013-03-11 14:10:02,023 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-37) [1cd48434] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 14:10:02,026 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-37) [218ee0a4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 14:10:02,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3415ab91 >2013-03-11 14:10:08,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 14:10:08,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-49) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:10:08,036 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 14:10:08,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 3415ab91 >2013-03-11 14:10:08,040 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [9a77ed0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:10:08,054 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [9a77ed0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 51e723aa >2013-03-11 14:10:08,054 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-49) [9a77ed0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 14:10:08,054 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-49) [9a77ed0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 51e723aa >2013-03-11 14:10:08,056 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-49) [9a77ed0] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 14:10:08,057 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [218ee0a4] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 14:10:08,066 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-37) [1848b65f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:10:08,068 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [1848b65f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 26612f03 >2013-03-11 14:10:08,069 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-37) [1848b65f] FINISH, SetVdsStatusVDSCommand, log id: 26612f03 >2013-03-11 14:10:08,074 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-37) [1848b65f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 14:10:08,098 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-37) [c9a448f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:10:08,100 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-37) [7db3b10e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:10:08,102 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-37) [7db3b10e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 14:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [9afc96e] Autorecovering 1 hosts >2013-03-11 14:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [9afc96e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [172407bc] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [172407bc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [172407bc] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3279d9ac >2013-03-11 14:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-50) [172407bc] FINISH, SetVdsStatusVDSCommand, log id: 3279d9ac >2013-03-11 14:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-50) [172407bc] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1b080797 >2013-03-11 14:15:00,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [172407bc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 69375ad5 >2013-03-11 14:15:00,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-50) [172407bc] FINISH, GetHardwareInfoVDSCommand, log id: 69375ad5 >2013-03-11 14:15:00,215 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-50) [64af887a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:15:00,218 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-50) [10085de2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:15:00,219 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-50) [10085de2] FINISH, ActivateVdsVDSCommand, log id: 1b080797 >2013-03-11 14:15:00,239 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-50) [10085de2] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:15:00,243 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [10085de2] Autorecovering 1 storage domains >2013-03-11 14:15:00,243 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-50) [10085de2] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 14:15:00,244 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [7c3cf50c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:15:00,244 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [7c3cf50c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:15 PM >2013-03-11 14:15:00,247 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-50) [7c3cf50c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:15 PM >2013-03-11 14:15:02,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [64f9c4dd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 255fe96a >2013-03-11 14:15:02,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-73) [64f9c4dd] FINISH, GetHardwareInfoVDSCommand, log id: 255fe96a >2013-03-11 14:15:02,390 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-73) [696d277a] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 14:15:02,393 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-73) [2b85c34b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 14:15:02,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 8080dab >2013-03-11 14:15:08,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 14:15:08,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:15:08,381 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 14:15:08,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 8080dab >2013-03-11 14:15:08,385 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [e89be6f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:15:08,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [e89be6f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 484cda12 >2013-03-11 14:15:08,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [e89be6f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 14:15:08,398 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [e89be6f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 484cda12 >2013-03-11 14:15:08,401 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [e89be6f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 14:15:08,401 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [2b85c34b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 14:15:08,410 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-73) [ce07d72] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:15:08,412 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [ce07d72] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 30bac6f >2013-03-11 14:15:08,413 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-73) [ce07d72] FINISH, SetVdsStatusVDSCommand, log id: 30bac6f >2013-03-11 14:15:08,418 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-73) [ce07d72] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 14:15:08,442 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-73) [618d93a2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:15:08,444 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-73) [4b624214] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:15:08,445 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-73) [4b624214] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 14:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [1b0f8e9f] Autorecovering 1 hosts >2013-03-11 14:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [1b0f8e9f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [582b5e9e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [582b5e9e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:20:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [582b5e9e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 333a7489 >2013-03-11 14:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [582b5e9e] FINISH, SetVdsStatusVDSCommand, log id: 333a7489 >2013-03-11 14:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [582b5e9e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: c177bbd >2013-03-11 14:20:00,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [582b5e9e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 25eddb49 >2013-03-11 14:20:00,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [582b5e9e] FINISH, GetHardwareInfoVDSCommand, log id: 25eddb49 >2013-03-11 14:20:00,188 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-84) [6aa8e12b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:20:00,191 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-84) [26251652] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:20:00,192 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [26251652] FINISH, ActivateVdsVDSCommand, log id: c177bbd >2013-03-11 14:20:00,211 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [26251652] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:20:00,212 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-84) [26251652] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 14:20:00,213 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [26251652] Autorecovering 1 storage domains >2013-03-11 14:20:00,213 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [26251652] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 14:20:00,214 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [1c72fa7a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:20:00,214 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [1c72fa7a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:20 PM >2013-03-11 14:20:00,217 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [1c72fa7a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:20 PM >2013-03-11 14:20:02,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [4f176f90] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 59f62bf2 >2013-03-11 14:20:02,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [4f176f90] FINISH, GetHardwareInfoVDSCommand, log id: 59f62bf2 >2013-03-11 14:20:02,739 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-91) [8321e9a] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 14:20:02,743 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-91) [4b7f39c1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 14:20:02,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 28bc1961 >2013-03-11 14:20:08,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 14:20:08,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:20:08,739 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 14:20:08,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 28bc1961 >2013-03-11 14:20:08,742 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [729abb43] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:20:08,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [729abb43] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4fac435c >2013-03-11 14:20:08,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [729abb43] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 14:20:08,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [729abb43] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4fac435c >2013-03-11 14:20:08,758 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [729abb43] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 14:20:08,758 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [4b7f39c1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 14:20:08,768 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-91) [4a1f5a12] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:20:08,769 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [4a1f5a12] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 24cc9014 >2013-03-11 14:20:08,770 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [4a1f5a12] FINISH, SetVdsStatusVDSCommand, log id: 24cc9014 >2013-03-11 14:20:08,776 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-91) [4a1f5a12] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 14:20:08,800 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-91) [4626a791] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:20:08,803 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-91) [9fe5421] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:20:08,805 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-91) [9fe5421] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 14:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [4101bdf6] Autorecovering 1 hosts >2013-03-11 14:25:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [4101bdf6] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:25:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [7722e6e7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:25:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [7722e6e7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:25:00,006 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [7722e6e7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 730bc4d9 >2013-03-11 14:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-69) [7722e6e7] FINISH, SetVdsStatusVDSCommand, log id: 730bc4d9 >2013-03-11 14:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [7722e6e7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1943d2c1 >2013-03-11 14:25:00,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [7722e6e7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3929567e >2013-03-11 14:25:00,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-69) [7722e6e7] FINISH, GetHardwareInfoVDSCommand, log id: 3929567e >2013-03-11 14:25:00,290 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-69) [430ba294] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:25:00,292 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-69) [282f77bc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:25:00,294 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-69) [282f77bc] FINISH, ActivateVdsVDSCommand, log id: 1943d2c1 >2013-03-11 14:25:00,313 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-69) [282f77bc] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:25:00,314 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-69) [282f77bc] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 14:25:00,315 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [282f77bc] Autorecovering 1 storage domains >2013-03-11 14:25:00,315 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-69) [282f77bc] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 14:25:00,316 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [26bcedab] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:25:00,316 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [26bcedab] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:25 PM >2013-03-11 14:25:00,319 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-69) [26bcedab] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:25 PM >2013-03-11 14:25:03,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [63789b1d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1505816e >2013-03-11 14:25:03,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [63789b1d] FINISH, GetHardwareInfoVDSCommand, log id: 1505816e >2013-03-11 14:25:03,091 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-11) [451b8956] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 14:25:03,095 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-11) [2335a2af] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 14:25:03,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1c470fda >2013-03-11 14:25:10,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 14:25:10,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:25:10,069 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 14:25:10,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 1c470fda >2013-03-11 14:25:10,073 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [71f36879] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:25:10,086 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [71f36879] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4aa2a66 >2013-03-11 14:25:10,086 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [71f36879] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 14:25:10,086 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [71f36879] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4aa2a66 >2013-03-11 14:25:10,088 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [71f36879] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 14:25:10,089 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [2335a2af] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 14:25:10,098 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-11) [2743907] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:25:10,100 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [2743907] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1509ba1a >2013-03-11 14:25:10,101 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [2743907] FINISH, SetVdsStatusVDSCommand, log id: 1509ba1a >2013-03-11 14:25:10,106 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-11) [2743907] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 14:25:10,130 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-11) [715cd757] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:25:10,133 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-11) [54bac27e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:25:10,134 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-11) [54bac27e] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 14:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [16cdcc0e] Autorecovering 1 hosts >2013-03-11 14:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [16cdcc0e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [4ab2bd06] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [4ab2bd06] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:30:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [4ab2bd06] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 26c72451 >2013-03-11 14:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [4ab2bd06] FINISH, SetVdsStatusVDSCommand, log id: 26c72451 >2013-03-11 14:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [4ab2bd06] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 666fe228 >2013-03-11 14:30:00,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [4ab2bd06] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6b0495a0 >2013-03-11 14:30:00,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [4ab2bd06] FINISH, GetHardwareInfoVDSCommand, log id: 6b0495a0 >2013-03-11 14:30:00,416 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [e5ca764] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:30:00,419 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [22586ef2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:30:00,420 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [22586ef2] FINISH, ActivateVdsVDSCommand, log id: 666fe228 >2013-03-11 14:30:00,441 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [22586ef2] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:30:00,442 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [22586ef2] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 14:30:00,443 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [22586ef2] Autorecovering 1 storage domains >2013-03-11 14:30:00,443 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [22586ef2] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 14:30:00,444 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [458caed7] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:30:00,444 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [458caed7] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:30 PM >2013-03-11 14:30:00,447 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [458caed7] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:30 PM >2013-03-11 14:30:01,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [789bca44] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6b2fa9ae >2013-03-11 14:30:01,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-47) [789bca44] FINISH, GetHardwareInfoVDSCommand, log id: 6b2fa9ae >2013-03-11 14:30:01,415 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-47) [33c41ec1] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 14:30:01,418 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-47) [3d94e673] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 14:30:01,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5d9893a1 >2013-03-11 14:30:08,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 14:30:08,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:30:08,475 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 14:30:08,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 5d9893a1 >2013-03-11 14:30:08,479 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [17416b44] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:30:08,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [17416b44] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: c245a8c >2013-03-11 14:30:08,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [17416b44] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 14:30:08,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [17416b44] FINISH, MarkPoolInReconstructModeVDSCommand, log id: c245a8c >2013-03-11 14:30:08,495 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [17416b44] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 14:30:08,496 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [3d94e673] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 14:30:08,505 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-47) [5a13afb0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:30:08,506 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [5a13afb0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 42853f3d >2013-03-11 14:30:08,508 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-47) [5a13afb0] FINISH, SetVdsStatusVDSCommand, log id: 42853f3d >2013-03-11 14:30:08,513 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-47) [5a13afb0] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 14:30:08,536 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-47) [a2f8af9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:30:08,539 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-47) [3d54158] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:30:08,540 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-47) [3d54158] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 14:30:33,890 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-90) [389116d0] Start refreshing all users data >2013-03-11 14:30:35,096 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [7eb8a541] Attempting to update VMs/Templates Ovf. >2013-03-11 14:30:35,097 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [7eb8a541] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 14:30:35,098 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [7eb8a541] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 14:30:35,098 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [7eb8a541] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 14:30:35,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [7eb8a541] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 14:30:35,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [7eb8a541] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 14:30:35,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-17) [7eb8a541] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 14:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [20184aaa] Autorecovering 1 hosts >2013-03-11 14:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [20184aaa] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-51) [218a2d42] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-51) [218a2d42] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [218a2d42] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2655e9e5 >2013-03-11 14:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-51) [218a2d42] FINISH, SetVdsStatusVDSCommand, log id: 2655e9e5 >2013-03-11 14:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-51) [218a2d42] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 66c38017 >2013-03-11 14:35:00,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [218a2d42] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: f687e69 >2013-03-11 14:35:00,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-51) [218a2d42] FINISH, GetHardwareInfoVDSCommand, log id: f687e69 >2013-03-11 14:35:00,308 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-51) [3ec41474] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:35:00,310 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-51) [66e9dc04] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:35:00,311 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-51) [66e9dc04] FINISH, ActivateVdsVDSCommand, log id: 66c38017 >2013-03-11 14:35:00,331 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-51) [66e9dc04] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:35:00,331 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-51) [66e9dc04] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 14:35:00,333 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [66e9dc04] Autorecovering 1 storage domains >2013-03-11 14:35:00,333 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-51) [66e9dc04] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 14:35:00,333 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-51) [1b748202] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:35:00,334 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-51) [1b748202] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:35 PM >2013-03-11 14:35:00,336 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-51) [1b748202] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:35 PM >2013-03-11 14:35:02,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [76c42510] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 67561dbd >2013-03-11 14:35:02,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [76c42510] FINISH, GetHardwareInfoVDSCommand, log id: 67561dbd >2013-03-11 14:35:02,827 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-45) [2976a94b] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 14:35:02,830 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-45) [185407a8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 14:35:02,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 70b75250 >2013-03-11 14:35:11,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 14:35:11,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:35:11,813 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 14:35:11,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 70b75250 >2013-03-11 14:35:11,817 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [67804a25] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:35:11,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [67804a25] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: f6b755a >2013-03-11 14:35:11,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [67804a25] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 14:35:11,831 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [67804a25] FINISH, MarkPoolInReconstructModeVDSCommand, log id: f6b755a >2013-03-11 14:35:11,833 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [67804a25] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 14:35:11,833 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [185407a8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 14:35:11,843 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-45) [6c40da25] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:35:11,844 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [6c40da25] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6088d3e2 >2013-03-11 14:35:11,845 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [6c40da25] FINISH, SetVdsStatusVDSCommand, log id: 6088d3e2 >2013-03-11 14:35:11,851 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-45) [6c40da25] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 14:35:11,876 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-45) [613f6ee8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:35:11,879 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-45) [54b6b7fc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:35:11,880 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-45) [54b6b7fc] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 14:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [4ed0941e] Autorecovering 1 hosts >2013-03-11 14:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [4ed0941e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [5462c69e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [5462c69e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [5462c69e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1985e162 >2013-03-11 14:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-58) [5462c69e] FINISH, SetVdsStatusVDSCommand, log id: 1985e162 >2013-03-11 14:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-58) [5462c69e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 68c8f98 >2013-03-11 14:40:00,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [5462c69e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 19adcbd3 >2013-03-11 14:40:00,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-58) [5462c69e] FINISH, GetHardwareInfoVDSCommand, log id: 19adcbd3 >2013-03-11 14:40:00,264 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-58) [425ea9c6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:40:00,266 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-58) [2770950d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:40:00,268 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-58) [2770950d] FINISH, ActivateVdsVDSCommand, log id: 68c8f98 >2013-03-11 14:40:00,288 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-58) [2770950d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:40:00,289 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-58) [2770950d] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 14:40:00,290 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [2770950d] Autorecovering 1 storage domains >2013-03-11 14:40:00,290 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-58) [2770950d] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 14:40:00,291 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [eb089ae] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:40:00,291 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [eb089ae] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:40 PM >2013-03-11 14:40:00,293 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-58) [eb089ae] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:40 PM >2013-03-11 14:40:00,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [620d05a4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7847e381 >2013-03-11 14:40:00,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [620d05a4] FINISH, GetHardwareInfoVDSCommand, log id: 7847e381 >2013-03-11 14:40:00,385 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-66) [4fc0f95b] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 14:40:00,389 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-66) [6157ee95] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 14:40:00,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6af44aec >2013-03-11 14:40:06,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 14:40:06,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:40:06,395 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 14:40:06,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 6af44aec >2013-03-11 14:40:06,399 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [21292b1e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:40:06,412 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [21292b1e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 703610d >2013-03-11 14:40:06,412 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [21292b1e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 14:40:06,412 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [21292b1e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 703610d >2013-03-11 14:40:06,414 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [21292b1e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 14:40:06,415 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [6157ee95] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 14:40:06,424 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-66) [4de9b8d8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:40:06,426 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [4de9b8d8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 58ce72ca >2013-03-11 14:40:06,427 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-66) [4de9b8d8] FINISH, SetVdsStatusVDSCommand, log id: 58ce72ca >2013-03-11 14:40:06,433 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-66) [4de9b8d8] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 14:40:06,458 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-66) [1c3173c8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:40:06,460 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-66) [fae2a73] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:40:06,462 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-66) [fae2a73] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 14:45:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [458caed7] Autorecovering 1 hosts >2013-03-11 14:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [458caed7] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [50b7949e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [50b7949e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [50b7949e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 17cc6b6e >2013-03-11 14:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [50b7949e] FINISH, SetVdsStatusVDSCommand, log id: 17cc6b6e >2013-03-11 14:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [50b7949e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 62779568 >2013-03-11 14:45:00,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [50b7949e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 21e94555 >2013-03-11 14:45:00,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [50b7949e] FINISH, GetHardwareInfoVDSCommand, log id: 21e94555 >2013-03-11 14:45:00,196 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [68e51601] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:45:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [68a066fd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:45:00,199 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [68a066fd] FINISH, ActivateVdsVDSCommand, log id: 62779568 >2013-03-11 14:45:00,219 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [68a066fd] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:45:00,219 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [68a066fd] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 14:45:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [68a066fd] Autorecovering 1 storage domains >2013-03-11 14:45:00,221 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [68a066fd] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 14:45:00,221 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [2289ffbe] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:45:00,221 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [2289ffbe] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:45 PM >2013-03-11 14:45:00,224 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [2289ffbe] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:45 PM >2013-03-11 14:45:00,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [23a8b68f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 341b6199 >2013-03-11 14:45:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [23a8b68f] FINISH, GetHardwareInfoVDSCommand, log id: 341b6199 >2013-03-11 14:45:00,752 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-7) [4087524c] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 14:45:00,755 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-7) [1d259b8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 14:45:00,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 50083a32 >2013-03-11 14:45:06,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 14:45:06,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:45:06,731 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 14:45:06,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 50083a32 >2013-03-11 14:45:06,734 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [d0476c1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:45:06,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [d0476c1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: f7c1dd2 >2013-03-11 14:45:06,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [d0476c1] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 14:45:06,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [d0476c1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: f7c1dd2 >2013-03-11 14:45:06,750 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [d0476c1] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 14:45:06,750 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [1d259b8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 14:45:06,760 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-7) [55322e51] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:45:06,761 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [55322e51] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 637e33d1 >2013-03-11 14:45:06,763 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [55322e51] FINISH, SetVdsStatusVDSCommand, log id: 637e33d1 >2013-03-11 14:45:06,768 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-7) [55322e51] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 14:45:06,792 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-7) [7f117117] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:45:06,794 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-7) [54dc2862] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:45:06,795 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) [54dc2862] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 14:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [406be437] Autorecovering 1 hosts >2013-03-11 14:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [406be437] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [406581e6] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [406581e6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [406581e6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 12859444 >2013-03-11 14:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [406581e6] FINISH, SetVdsStatusVDSCommand, log id: 12859444 >2013-03-11 14:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-92) [406581e6] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 662a5866 >2013-03-11 14:50:00,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [406581e6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 724317fb >2013-03-11 14:50:00,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [406581e6] FINISH, GetHardwareInfoVDSCommand, log id: 724317fb >2013-03-11 14:50:00,296 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-92) [af692f4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:50:00,298 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-92) [1327d23b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:50:00,300 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-92) [1327d23b] FINISH, ActivateVdsVDSCommand, log id: 662a5866 >2013-03-11 14:50:00,320 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [1327d23b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:50:00,321 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-92) [1327d23b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 14:50:00,323 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [1327d23b] Autorecovering 1 storage domains >2013-03-11 14:50:00,323 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [1327d23b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 14:50:00,323 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [7e5f59d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:50:00,323 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [7e5f59d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:50 PM >2013-03-11 14:50:00,326 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [7e5f59d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:50 PM >2013-03-11 14:50:01,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [296830f7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3417d1d6 >2013-03-11 14:50:01,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-26) [296830f7] FINISH, GetHardwareInfoVDSCommand, log id: 3417d1d6 >2013-03-11 14:50:01,100 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-26) [42b67f7a] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 14:50:01,104 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-26) [2da35bf0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 14:50:01,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: a6bf8cb >2013-03-11 14:50:07,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 14:50:07,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:50:07,081 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 14:50:07,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: a6bf8cb >2013-03-11 14:50:07,086 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [2f73b923] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:50:07,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [2f73b923] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 290ede6c >2013-03-11 14:50:07,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [2f73b923] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 14:50:07,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [2f73b923] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 290ede6c >2013-03-11 14:50:07,102 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [2f73b923] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 14:50:07,102 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [2da35bf0] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 14:50:07,112 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-26) [b247fec] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:50:07,114 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [b247fec] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1970e8a2 >2013-03-11 14:50:07,115 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-26) [b247fec] FINISH, SetVdsStatusVDSCommand, log id: 1970e8a2 >2013-03-11 14:50:07,121 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-26) [b247fec] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 14:50:07,146 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-26) [246152e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:50:07,149 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-26) [31071611] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:50:07,150 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-26) [31071611] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 14:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [783825bc] Autorecovering 1 hosts >2013-03-11 14:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [783825bc] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [4ff16b7f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [4ff16b7f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [4ff16b7f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 788e4c20 >2013-03-11 14:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [4ff16b7f] FINISH, SetVdsStatusVDSCommand, log id: 788e4c20 >2013-03-11 14:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-93) [4ff16b7f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 174dc0af >2013-03-11 14:55:00,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [4ff16b7f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 43204e5a >2013-03-11 14:55:00,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [4ff16b7f] FINISH, GetHardwareInfoVDSCommand, log id: 43204e5a >2013-03-11 14:55:00,207 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [52aa2056] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:55:00,209 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [cc906f2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:55:00,211 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-93) [cc906f2] FINISH, ActivateVdsVDSCommand, log id: 174dc0af >2013-03-11 14:55:00,230 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-93) [cc906f2] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 14:55:00,231 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-93) [cc906f2] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 14:55:00,232 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [cc906f2] Autorecovering 1 storage domains >2013-03-11 14:55:00,232 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-93) [cc906f2] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 14:55:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [7663f422] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:55:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [7663f422] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 2:55 PM >2013-03-11 14:55:00,236 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-93) [7663f422] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 2:55 PM >2013-03-11 14:55:01,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [1effb5f1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 17d8994b >2013-03-11 14:55:01,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-2) [1effb5f1] FINISH, GetHardwareInfoVDSCommand, log id: 17d8994b >2013-03-11 14:55:01,447 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-2) [4593159d] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 14:55:01,450 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-2) [469f0e93] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 14:55:01,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 43b6d9a1 >2013-03-11 14:55:07,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 14:55:07,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 14:55:07,473 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 14:55:07,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 43b6d9a1 >2013-03-11 14:55:07,477 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [56cf6c0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 14:55:07,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [56cf6c0] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7eb25e4b >2013-03-11 14:55:07,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [56cf6c0] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 14:55:07,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [56cf6c0] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7eb25e4b >2013-03-11 14:55:07,492 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [56cf6c0] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 14:55:07,493 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [469f0e93] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 14:55:07,502 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-2) [5be5c18a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:55:07,504 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [5be5c18a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1c757639 >2013-03-11 14:55:07,505 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-2) [5be5c18a] FINISH, SetVdsStatusVDSCommand, log id: 1c757639 >2013-03-11 14:55:07,511 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-2) [5be5c18a] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 14:55:07,535 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-2) [14b78f19] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:55:07,537 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-2) [5ba55ac3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 14:55:07,538 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-2) [5ba55ac3] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 15:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [5e8f40cb] Autorecovering 1 hosts >2013-03-11 15:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [5e8f40cb] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [29eda2e0] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [29eda2e0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:00:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [29eda2e0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2730e55a >2013-03-11 15:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-20) [29eda2e0] FINISH, SetVdsStatusVDSCommand, log id: 2730e55a >2013-03-11 15:00:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-20) [29eda2e0] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2baa5f3f >2013-03-11 15:00:00,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [29eda2e0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5c64e869 >2013-03-11 15:00:00,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-20) [29eda2e0] FINISH, GetHardwareInfoVDSCommand, log id: 5c64e869 >2013-03-11 15:00:00,212 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-20) [6afa13f2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:00:00,215 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-20) [6d18ef45] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:00:00,217 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-20) [6d18ef45] FINISH, ActivateVdsVDSCommand, log id: 2baa5f3f >2013-03-11 15:00:00,236 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-20) [6d18ef45] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:00:00,237 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-20) [6d18ef45] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 15:00:00,238 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [6d18ef45] Autorecovering 1 storage domains >2013-03-11 15:00:00,238 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-20) [6d18ef45] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 15:00:00,239 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [713599e7] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:00:00,239 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [713599e7] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:00 PM >2013-03-11 15:00:00,242 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-20) [713599e7] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:00 PM >2013-03-11 15:00:01,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [452f63bc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2a383de4 >2013-03-11 15:00:01,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [452f63bc] FINISH, GetHardwareInfoVDSCommand, log id: 2a383de4 >2013-03-11 15:00:01,824 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-94) [5ccd0cfe] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 15:00:01,827 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-94) [4dd942ea] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 15:00:01,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2524bca1 >2013-03-11 15:00:07,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 15:00:07,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:00:07,872 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 15:00:07,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 2524bca1 >2013-03-11 15:00:07,876 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [467b57e1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:00:07,890 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [467b57e1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2d7bc68c >2013-03-11 15:00:07,890 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [467b57e1] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 15:00:07,890 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [467b57e1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2d7bc68c >2013-03-11 15:00:07,892 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [467b57e1] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 15:00:07,893 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [4dd942ea] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 15:00:07,903 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-94) [6f2fdaa1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:00:07,904 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [6f2fdaa1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: fcd9316 >2013-03-11 15:00:07,905 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [6f2fdaa1] FINISH, SetVdsStatusVDSCommand, log id: fcd9316 >2013-03-11 15:00:07,911 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-94) [6f2fdaa1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 15:00:07,935 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [524e6123] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:00:07,937 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [483b71a0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:00:07,939 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-94) [483b71a0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 15:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [3652e5ee] Autorecovering 1 hosts >2013-03-11 15:05:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [3652e5ee] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:05:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [dc6614b] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:05:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [dc6614b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [dc6614b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 35472577 >2013-03-11 15:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [dc6614b] FINISH, SetVdsStatusVDSCommand, log id: 35472577 >2013-03-11 15:05:00,010 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-75) [dc6614b] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 514b9d91 >2013-03-11 15:05:00,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [dc6614b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 63b9ab21 >2013-03-11 15:05:00,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [dc6614b] FINISH, GetHardwareInfoVDSCommand, log id: 63b9ab21 >2013-03-11 15:05:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [2ff3f5e2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:05:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [5bdd53cf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:05:00,206 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-75) [5bdd53cf] FINISH, ActivateVdsVDSCommand, log id: 514b9d91 >2013-03-11 15:05:00,227 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-75) [5bdd53cf] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:05:00,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-75) [5bdd53cf] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 15:05:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [5bdd53cf] Autorecovering 1 storage domains >2013-03-11 15:05:00,229 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-75) [5bdd53cf] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 15:05:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [53af3509] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:05:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [53af3509] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:05 PM >2013-03-11 15:05:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-75) [53af3509] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:05 PM >2013-03-11 15:05:02,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [22b4a4bf] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6094cf75 >2013-03-11 15:05:02,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [22b4a4bf] FINISH, GetHardwareInfoVDSCommand, log id: 6094cf75 >2013-03-11 15:05:02,265 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-70) [3bf3057e] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 15:05:02,269 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-70) [42c1d0ae] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 15:05:02,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 13ce0a84 >2013-03-11 15:05:09,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 15:05:09,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:05:09,277 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 15:05:09,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 13ce0a84 >2013-03-11 15:05:09,281 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [798da7fb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:05:09,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [798da7fb] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3e8ea5cc >2013-03-11 15:05:09,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [798da7fb] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 15:05:09,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [798da7fb] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3e8ea5cc >2013-03-11 15:05:09,300 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [42c1d0ae] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 15:05:09,310 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-70) [f6c0763] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:05:09,311 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [f6c0763] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 336549c7 >2013-03-11 15:05:09,312 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [f6c0763] FINISH, SetVdsStatusVDSCommand, log id: 336549c7 >2013-03-11 15:05:09,318 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-70) [f6c0763] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 15:05:09,341 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-70) [3f90dff3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:05:09,344 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-70) [518e1018] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:05:09,345 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-70) [518e1018] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 15:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [248e1751] Autorecovering 1 hosts >2013-03-11 15:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [248e1751] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-56) [5e5f6c0e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-56) [5e5f6c0e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [5e5f6c0e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7e6bdf9c >2013-03-11 15:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [5e5f6c0e] FINISH, SetVdsStatusVDSCommand, log id: 7e6bdf9c >2013-03-11 15:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-56) [5e5f6c0e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 63d8aeac >2013-03-11 15:10:00,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [5e5f6c0e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 394123cf >2013-03-11 15:10:00,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [5e5f6c0e] FINISH, GetHardwareInfoVDSCommand, log id: 394123cf >2013-03-11 15:10:00,259 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-56) [25ef181c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:10:00,261 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-56) [1a01abb1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:10:00,262 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-56) [1a01abb1] FINISH, ActivateVdsVDSCommand, log id: 63d8aeac >2013-03-11 15:10:00,282 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-56) [1a01abb1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:10:00,283 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-56) [1a01abb1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 15:10:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [1a01abb1] Autorecovering 1 storage domains >2013-03-11 15:10:00,284 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-56) [1a01abb1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 15:10:00,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-56) [6ba4602a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:10:00,285 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-56) [6ba4602a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:10 PM >2013-03-11 15:10:00,287 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-56) [6ba4602a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:10 PM >2013-03-11 15:10:00,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [35412d54] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 32d9f78 >2013-03-11 15:10:00,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-82) [35412d54] FINISH, GetHardwareInfoVDSCommand, log id: 32d9f78 >2013-03-11 15:10:00,628 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-82) [ba8d597] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 15:10:00,632 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-82) [344e7919] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 15:10:00,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 3c656981 >2013-03-11 15:10:05,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 15:10:05,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:10:05,628 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 15:10:05,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 3c656981 >2013-03-11 15:10:05,631 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [1368b7ff] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:10:05,644 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [1368b7ff] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4ba34a9b >2013-03-11 15:10:05,644 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [1368b7ff] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 15:10:05,644 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [1368b7ff] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4ba34a9b >2013-03-11 15:10:05,647 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [1368b7ff] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 15:10:05,647 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [344e7919] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 15:10:05,662 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-82) [680bb3a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:10:05,664 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [680bb3a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 39ab1ef >2013-03-11 15:10:05,665 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-82) [680bb3a] FINISH, SetVdsStatusVDSCommand, log id: 39ab1ef >2013-03-11 15:10:05,670 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-82) [680bb3a] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 15:10:05,693 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-82) [1900e2ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:10:05,696 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-82) [30754ac2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:10:05,697 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-82) [30754ac2] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 15:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [1d98e869] Autorecovering 1 hosts >2013-03-11 15:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [1d98e869] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-25) [6a307aab] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-25) [6a307aab] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:15:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [6a307aab] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: a5fd074 >2013-03-11 15:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [6a307aab] FINISH, SetVdsStatusVDSCommand, log id: a5fd074 >2013-03-11 15:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-25) [6a307aab] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 44be8537 >2013-03-11 15:15:00,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [6a307aab] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7e69d0de >2013-03-11 15:15:00,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [6a307aab] FINISH, GetHardwareInfoVDSCommand, log id: 7e69d0de >2013-03-11 15:15:00,405 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-25) [432bab20] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:15:00,408 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-25) [f14e265] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:15:00,410 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-25) [f14e265] FINISH, ActivateVdsVDSCommand, log id: 44be8537 >2013-03-11 15:15:00,429 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-25) [f14e265] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:15:00,430 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-25) [f14e265] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 15:15:00,431 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [f14e265] Autorecovering 1 storage domains >2013-03-11 15:15:00,431 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-25) [f14e265] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 15:15:00,432 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-25) [4e770f42] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:15:00,432 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-25) [4e770f42] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:15 PM >2013-03-11 15:15:00,435 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-25) [4e770f42] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:15 PM >2013-03-11 15:15:02,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [2289ffbe] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1a29aa0b >2013-03-11 15:15:02,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [2289ffbe] FINISH, GetHardwareInfoVDSCommand, log id: 1a29aa0b >2013-03-11 15:15:02,981 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-36) [287959d8] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 15:15:02,985 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-36) [53d206a4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 15:15:02,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4b99c1d3 >2013-03-11 15:15:09,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 15:15:09,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:15:09,960 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 15:15:09,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 4b99c1d3 >2013-03-11 15:15:09,963 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [7340c7b8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:15:09,977 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [7340c7b8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7a5f2c3 >2013-03-11 15:15:09,977 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [7340c7b8] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 15:15:09,977 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [7340c7b8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7a5f2c3 >2013-03-11 15:15:09,980 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [7340c7b8] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 15:15:09,980 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [53d206a4] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 15:15:09,990 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-36) [3962593b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:15:09,992 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [3962593b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 77f37173 >2013-03-11 15:15:09,993 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [3962593b] FINISH, SetVdsStatusVDSCommand, log id: 77f37173 >2013-03-11 15:15:09,998 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [3962593b] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 15:15:10,022 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [13290875] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:15:10,028 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [1009adde] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:15:10,030 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-36) [1009adde] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 15:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [271504c2] Autorecovering 1 hosts >2013-03-11 15:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [271504c2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-86) [494959c0] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-86) [494959c0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [494959c0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 575c7ce6 >2013-03-11 15:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-86) [494959c0] FINISH, SetVdsStatusVDSCommand, log id: 575c7ce6 >2013-03-11 15:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-86) [494959c0] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7df08704 >2013-03-11 15:20:00,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [494959c0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 51c48d5d >2013-03-11 15:20:00,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-86) [494959c0] FINISH, GetHardwareInfoVDSCommand, log id: 51c48d5d >2013-03-11 15:20:00,377 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-86) [23ec3489] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:20:00,380 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-86) [35883c4e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:20:00,381 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-86) [35883c4e] FINISH, ActivateVdsVDSCommand, log id: 7df08704 >2013-03-11 15:20:00,400 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-86) [35883c4e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:20:00,401 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-86) [35883c4e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 15:20:00,402 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [35883c4e] Autorecovering 1 storage domains >2013-03-11 15:20:00,402 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-86) [35883c4e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 15:20:00,403 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-86) [61d9874c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:20:00,403 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-86) [61d9874c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:20 PM >2013-03-11 15:20:00,406 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-86) [61d9874c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:20 PM >2013-03-11 15:20:01,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [18568a9b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4e6e417d >2013-03-11 15:20:01,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-99) [18568a9b] FINISH, GetHardwareInfoVDSCommand, log id: 4e6e417d >2013-03-11 15:20:01,324 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-99) [4e2995b2] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 15:20:01,327 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-99) [4b0a3040] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 15:20:01,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1adcb13c >2013-03-11 15:20:08,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 15:20:08,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:20:08,318 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 15:20:08,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 1adcb13c >2013-03-11 15:20:08,322 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [54c2ed28] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:20:08,335 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [54c2ed28] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2e346406 >2013-03-11 15:20:08,335 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [54c2ed28] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 15:20:08,335 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [54c2ed28] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2e346406 >2013-03-11 15:20:08,337 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [54c2ed28] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 15:20:08,338 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [4b0a3040] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 15:20:08,347 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-99) [554c87e1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:20:08,349 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [554c87e1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1f85752 >2013-03-11 15:20:08,350 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-99) [554c87e1] FINISH, SetVdsStatusVDSCommand, log id: 1f85752 >2013-03-11 15:20:08,355 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-99) [554c87e1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 15:20:08,381 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-99) [29a6f906] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:20:08,384 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-99) [7ead886a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:20:08,385 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-99) [7ead886a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 15:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [7cb642fe] Autorecovering 1 hosts >2013-03-11 15:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [7cb642fe] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [277c874e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [277c874e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:25:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [277c874e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 27a10b24 >2013-03-11 15:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [277c874e] FINISH, SetVdsStatusVDSCommand, log id: 27a10b24 >2013-03-11 15:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [277c874e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3e0a7da >2013-03-11 15:25:00,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [277c874e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 48815754 >2013-03-11 15:25:00,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [277c874e] FINISH, GetHardwareInfoVDSCommand, log id: 48815754 >2013-03-11 15:25:00,533 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-79) [3d399d7d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:25:00,535 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-79) [5057d9d1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:25:00,536 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-79) [5057d9d1] FINISH, ActivateVdsVDSCommand, log id: 3e0a7da >2013-03-11 15:25:00,556 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-79) [5057d9d1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:25:00,557 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-79) [5057d9d1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 15:25:00,558 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [5057d9d1] Autorecovering 1 storage domains >2013-03-11 15:25:00,558 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-79) [5057d9d1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 15:25:00,559 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [12ed4689] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:25:00,559 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [12ed4689] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:25 PM >2013-03-11 15:25:00,562 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-79) [12ed4689] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:25 PM >2013-03-11 15:25:02,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [3fce8e6e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 324e0986 >2013-03-11 15:25:02,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [3fce8e6e] FINISH, GetHardwareInfoVDSCommand, log id: 324e0986 >2013-03-11 15:25:02,671 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-42) [1bf7bff7] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 15:25:02,675 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-42) [51b7b8b4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 15:25:02,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 435c8743 >2013-03-11 15:25:08,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 15:25:08,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:25:08,676 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 15:25:08,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 435c8743 >2013-03-11 15:25:08,680 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [73c0b8c1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:25:08,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [73c0b8c1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6955de2a >2013-03-11 15:25:08,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [73c0b8c1] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 15:25:08,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [73c0b8c1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6955de2a >2013-03-11 15:25:08,696 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [73c0b8c1] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 15:25:08,696 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [51b7b8b4] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 15:25:08,706 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-42) [46f976f3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:25:08,708 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [46f976f3] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 47ffc8d8 >2013-03-11 15:25:08,709 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [46f976f3] FINISH, SetVdsStatusVDSCommand, log id: 47ffc8d8 >2013-03-11 15:25:08,715 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-42) [46f976f3] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 15:25:08,739 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-42) [4d17c5e7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:25:08,742 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-42) [14ed8f06] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:25:08,743 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-42) [14ed8f06] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 15:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [479b339f] Autorecovering 1 hosts >2013-03-11 15:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [479b339f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-23) [786bbdb4] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-23) [786bbdb4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:30:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-23) [786bbdb4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1aa1d317 >2013-03-11 15:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-23) [786bbdb4] FINISH, SetVdsStatusVDSCommand, log id: 1aa1d317 >2013-03-11 15:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-23) [786bbdb4] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 43c1e38b >2013-03-11 15:30:00,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-23) [786bbdb4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2bd3a95f >2013-03-11 15:30:00,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-23) [786bbdb4] FINISH, GetHardwareInfoVDSCommand, log id: 2bd3a95f >2013-03-11 15:30:00,104 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-23) [2e2351cc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:30:00,106 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-23) [5f2af8e2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:30:00,107 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-23) [5f2af8e2] FINISH, ActivateVdsVDSCommand, log id: 43c1e38b >2013-03-11 15:30:00,126 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-23) [5f2af8e2] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:30:00,127 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-23) [5f2af8e2] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 15:30:00,129 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [5f2af8e2] Autorecovering 1 storage domains >2013-03-11 15:30:00,129 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-23) [5f2af8e2] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 15:30:00,129 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-23) [6521f7b4] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:30:00,129 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-23) [6521f7b4] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:30 PM >2013-03-11 15:30:00,132 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-23) [6521f7b4] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:30 PM >2013-03-11 15:30:02,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [547d49ab] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 19c57657 >2013-03-11 15:30:02,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [547d49ab] FINISH, GetHardwareInfoVDSCommand, log id: 19c57657 >2013-03-11 15:30:03,032 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-72) [6935486b] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 15:30:03,035 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-72) [6587ca36] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 15:30:03,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2456f4d7 >2013-03-11 15:30:08,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 15:30:08,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:30:08,058 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 15:30:08,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2456f4d7 >2013-03-11 15:30:08,062 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [3abe2501] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:30:08,076 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [3abe2501] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 26181134 >2013-03-11 15:30:08,076 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [3abe2501] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 15:30:08,076 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [3abe2501] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 26181134 >2013-03-11 15:30:08,078 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [3abe2501] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 15:30:08,079 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [6587ca36] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 15:30:08,089 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-72) [33309d6c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:30:08,090 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [33309d6c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6b95ab7c >2013-03-11 15:30:08,091 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [33309d6c] FINISH, SetVdsStatusVDSCommand, log id: 6b95ab7c >2013-03-11 15:30:08,097 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [33309d6c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 15:30:08,121 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [351af9f6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:30:08,124 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [2e4e7034] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:30:08,125 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-72) [2e4e7034] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 15:30:33,892 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-65) [1fd27d46] Start refreshing all users data >2013-03-11 15:30:35,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-69) [26bcedab] Attempting to update VMs/Templates Ovf. >2013-03-11 15:30:35,100 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-69) [26bcedab] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 15:30:35,101 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-69) [26bcedab] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 15:30:35,101 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-69) [26bcedab] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 15:30:35,101 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-69) [26bcedab] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 15:30:35,101 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-69) [26bcedab] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 15:30:35,102 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-69) [26bcedab] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 15:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [d9efe9e] Autorecovering 1 hosts >2013-03-11 15:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [d9efe9e] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [5b8690c9] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [5b8690c9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [5b8690c9] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7a34e39d >2013-03-11 15:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [5b8690c9] FINISH, SetVdsStatusVDSCommand, log id: 7a34e39d >2013-03-11 15:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-19) [5b8690c9] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5e06782a >2013-03-11 15:35:00,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [5b8690c9] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 413ff9d3 >2013-03-11 15:35:00,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [5b8690c9] FINISH, GetHardwareInfoVDSCommand, log id: 413ff9d3 >2013-03-11 15:35:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-19) [5b7bfa85] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:35:00,207 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-19) [1e2b9f16] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:35:00,208 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-19) [1e2b9f16] FINISH, ActivateVdsVDSCommand, log id: 5e06782a >2013-03-11 15:35:00,228 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [1e2b9f16] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:35:00,229 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-19) [1e2b9f16] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 15:35:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [1e2b9f16] Autorecovering 1 storage domains >2013-03-11 15:35:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [1e2b9f16] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 15:35:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [32acbdc8] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:35:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [32acbdc8] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:35 PM >2013-03-11 15:35:00,233 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [32acbdc8] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:35 PM >2013-03-11 15:35:02,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-3) [3c5897be] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3060462c >2013-03-11 15:35:02,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-3) [3c5897be] FINISH, GetHardwareInfoVDSCommand, log id: 3060462c >2013-03-11 15:35:02,412 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-3) [33988c04] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 15:35:02,416 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-3) [d74cece] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 15:35:02,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2f0c4e7c >2013-03-11 15:35:07,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 15:35:07,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:35:07,733 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 15:35:07,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2f0c4e7c >2013-03-11 15:35:07,737 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [6a695f19] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:35:07,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6a695f19] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 18c354d4 >2013-03-11 15:35:07,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [6a695f19] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 15:35:07,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6a695f19] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 18c354d4 >2013-03-11 15:35:07,753 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [6a695f19] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 15:35:07,754 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [d74cece] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 15:35:07,764 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-3) [4c4804ba] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:35:07,765 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-3) [4c4804ba] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 71362d33 >2013-03-11 15:35:07,766 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-3) [4c4804ba] FINISH, SetVdsStatusVDSCommand, log id: 71362d33 >2013-03-11 15:35:07,772 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-3) [4c4804ba] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 15:35:07,797 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-3) [4a3feef6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:35:07,800 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-3) [3864a1c4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:35:07,801 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-3) [3864a1c4] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 15:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [2f602012] Autorecovering 1 hosts >2013-03-11 15:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [2f602012] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [6c2aaa6f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [6c2aaa6f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [6c2aaa6f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: a359c24 >2013-03-11 15:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [6c2aaa6f] FINISH, SetVdsStatusVDSCommand, log id: a359c24 >2013-03-11 15:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [6c2aaa6f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 53a677fb >2013-03-11 15:40:00,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [6c2aaa6f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4ba7edae >2013-03-11 15:40:00,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [6c2aaa6f] FINISH, GetHardwareInfoVDSCommand, log id: 4ba7edae >2013-03-11 15:40:00,409 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [2440563d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:40:00,412 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [796bd1c0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:40:00,413 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-43) [796bd1c0] FINISH, ActivateVdsVDSCommand, log id: 53a677fb >2013-03-11 15:40:00,433 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-43) [796bd1c0] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:40:00,434 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [796bd1c0] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 15:40:00,435 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [796bd1c0] Autorecovering 1 storage domains >2013-03-11 15:40:00,435 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-43) [796bd1c0] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 15:40:00,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [35f1011e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:40:00,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [35f1011e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:40 PM >2013-03-11 15:40:00,439 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-43) [35f1011e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:40 PM >2013-03-11 15:40:02,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [7663f422] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 78daa0c3 >2013-03-11 15:40:02,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-93) [7663f422] FINISH, GetHardwareInfoVDSCommand, log id: 78daa0c3 >2013-03-11 15:40:02,094 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-93) [38d71db6] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 15:40:02,098 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-93) [5c300b73] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 15:40:02,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4d6cbbdb >2013-03-11 15:40:11,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 15:40:11,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:40:11,113 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 15:40:11,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 4d6cbbdb >2013-03-11 15:40:11,117 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [330f7fd3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:40:11,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [330f7fd3] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5a72708e >2013-03-11 15:40:11,133 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [330f7fd3] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 15:40:11,133 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [330f7fd3] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5a72708e >2013-03-11 15:40:11,135 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [330f7fd3] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 15:40:11,135 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [5c300b73] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 15:40:11,146 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-93) [278aa2a4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:40:11,147 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [278aa2a4] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4f6f28f9 >2013-03-11 15:40:11,148 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-93) [278aa2a4] FINISH, SetVdsStatusVDSCommand, log id: 4f6f28f9 >2013-03-11 15:40:11,154 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-93) [278aa2a4] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 15:40:11,209 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-93) [58948059] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:40:11,212 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-93) [39f376f4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:40:11,213 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-93) [39f376f4] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 15:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [5331bd9a] Autorecovering 1 hosts >2013-03-11 15:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [5331bd9a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [199ffb4f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [199ffb4f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [199ffb4f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 67eba13e >2013-03-11 15:45:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [199ffb4f] FINISH, SetVdsStatusVDSCommand, log id: 67eba13e >2013-03-11 15:45:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [199ffb4f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1affcda >2013-03-11 15:45:00,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [199ffb4f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 68e115e1 >2013-03-11 15:45:00,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [199ffb4f] FINISH, GetHardwareInfoVDSCommand, log id: 68e115e1 >2013-03-11 15:45:00,414 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [3c0e0e26] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:45:00,416 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [132dba64] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:45:00,418 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [132dba64] FINISH, ActivateVdsVDSCommand, log id: 1affcda >2013-03-11 15:45:00,437 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [132dba64] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:45:00,438 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-8) [132dba64] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 15:45:00,439 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [132dba64] Autorecovering 1 storage domains >2013-03-11 15:45:00,439 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [132dba64] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 15:45:00,440 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [376e74e2] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:45:00,440 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [376e74e2] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:45 PM >2013-03-11 15:45:00,443 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [376e74e2] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:45 PM >2013-03-11 15:45:02,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [4e770f42] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 52f7d378 >2013-03-11 15:45:02,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-25) [4e770f42] FINISH, GetHardwareInfoVDSCommand, log id: 52f7d378 >2013-03-11 15:45:02,506 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-25) [1f3fe50] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 15:45:02,509 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-25) [74ece30] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 15:45:02,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 372c2a27 >2013-03-11 15:45:09,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 15:45:09,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:45:09,483 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 15:45:09,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 372c2a27 >2013-03-11 15:45:09,487 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [229c336e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:45:09,501 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [229c336e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7a58e551 >2013-03-11 15:45:09,501 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [229c336e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 15:45:09,501 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [229c336e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7a58e551 >2013-03-11 15:45:09,503 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [229c336e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 15:45:09,504 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [74ece30] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 15:45:09,513 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-25) [1d97afb5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:45:09,515 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [1d97afb5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 71b58a7 >2013-03-11 15:45:09,516 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-25) [1d97afb5] FINISH, SetVdsStatusVDSCommand, log id: 71b58a7 >2013-03-11 15:45:09,528 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-25) [1d97afb5] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 15:45:09,565 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-25) [11193488] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:45:09,567 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-25) [4ea112ba] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:45:09,569 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-25) [4ea112ba] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 15:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [6a863ebd] Autorecovering 1 hosts >2013-03-11 15:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [6a863ebd] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [1c6cfc8f] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [1c6cfc8f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [1c6cfc8f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2bf631ee >2013-03-11 15:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [1c6cfc8f] FINISH, SetVdsStatusVDSCommand, log id: 2bf631ee >2013-03-11 15:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-46) [1c6cfc8f] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 72e1546c >2013-03-11 15:50:00,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [1c6cfc8f] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7eea459d >2013-03-11 15:50:00,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [1c6cfc8f] FINISH, GetHardwareInfoVDSCommand, log id: 7eea459d >2013-03-11 15:50:00,394 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-46) [7d6b9fa3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:50:00,396 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-46) [291bb54c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:50:00,398 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-46) [291bb54c] FINISH, ActivateVdsVDSCommand, log id: 72e1546c >2013-03-11 15:50:00,417 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [291bb54c] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:50:00,418 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-46) [291bb54c] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 15:50:00,419 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [291bb54c] Autorecovering 1 storage domains >2013-03-11 15:50:00,419 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [291bb54c] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 15:50:00,420 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [755df19b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:50:00,420 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [755df19b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:50 PM >2013-03-11 15:50:00,423 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [755df19b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:50 PM >2013-03-11 15:50:00,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [7cf91ca0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 167da600 >2013-03-11 15:50:00,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-95) [7cf91ca0] FINISH, GetHardwareInfoVDSCommand, log id: 167da600 >2013-03-11 15:50:00,854 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-95) [527de8a9] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 15:50:00,858 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-95) [2e4c834c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 15:50:00,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: e6baccc >2013-03-11 15:50:07,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 15:50:07,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:50:07,852 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 15:50:07,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: e6baccc >2013-03-11 15:50:07,856 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [2ef63c8d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:50:07,869 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [2ef63c8d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 778403d8 >2013-03-11 15:50:07,870 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [2ef63c8d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 15:50:07,870 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [2ef63c8d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 778403d8 >2013-03-11 15:50:07,872 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [2ef63c8d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 15:50:07,872 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [2e4c834c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 15:50:07,882 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-95) [5db72ff6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:50:07,883 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [5db72ff6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2881cc72 >2013-03-11 15:50:07,884 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-95) [5db72ff6] FINISH, SetVdsStatusVDSCommand, log id: 2881cc72 >2013-03-11 15:50:07,893 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-95) [5db72ff6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 15:50:07,918 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-95) [7c467377] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:50:07,921 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-95) [7777b5f1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:50:07,922 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-95) [7777b5f1] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 15:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [4e1c2020] Autorecovering 1 hosts >2013-03-11 15:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [4e1c2020] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [669d07ed] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [669d07ed] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:55:00,041 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [669d07ed] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3adb296c >2013-03-11 15:55:00,042 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [669d07ed] FINISH, SetVdsStatusVDSCommand, log id: 3adb296c >2013-03-11 15:55:00,045 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-57) [669d07ed] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7757c50d >2013-03-11 15:55:00,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [669d07ed] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7bab6408 >2013-03-11 15:55:00,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [669d07ed] FINISH, GetHardwareInfoVDSCommand, log id: 7bab6408 >2013-03-11 15:55:00,322 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-57) [1ad807bf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:55:00,325 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-57) [1abf4c9b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:55:00,326 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-57) [1abf4c9b] FINISH, ActivateVdsVDSCommand, log id: 7757c50d >2013-03-11 15:55:00,346 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-57) [1abf4c9b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 15:55:00,347 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-57) [1abf4c9b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 15:55:00,348 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [1abf4c9b] Autorecovering 1 storage domains >2013-03-11 15:55:00,348 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-57) [1abf4c9b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 15:55:00,349 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [d9c8cd4] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:55:00,349 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [d9c8cd4] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 3:55 PM >2013-03-11 15:55:00,351 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-57) [d9c8cd4] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 3:55 PM >2013-03-11 15:55:02,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [35f1011e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4511d542 >2013-03-11 15:55:02,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-43) [35f1011e] FINISH, GetHardwareInfoVDSCommand, log id: 4511d542 >2013-03-11 15:55:02,217 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-43) [5007419c] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 15:55:02,221 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-43) [20c83808] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 15:55:02,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 16157714 >2013-03-11 15:55:09,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 15:55:09,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 15:55:09,210 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 15:55:09,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 16157714 >2013-03-11 15:55:09,214 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [78b778f6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 15:55:09,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [78b778f6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7ee5cb81 >2013-03-11 15:55:09,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [78b778f6] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 15:55:09,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [78b778f6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7ee5cb81 >2013-03-11 15:55:09,229 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [78b778f6] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 15:55:09,230 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [20c83808] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 15:55:09,240 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-43) [cf8e322] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:55:09,242 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [cf8e322] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3aa503e8 >2013-03-11 15:55:09,243 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-43) [cf8e322] FINISH, SetVdsStatusVDSCommand, log id: 3aa503e8 >2013-03-11 15:55:09,253 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-43) [cf8e322] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 15:55:09,306 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-43) [174435a5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:55:09,309 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-43) [4be55521] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 15:55:09,311 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-43) [4be55521] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 16:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [1cea420a] Autorecovering 1 hosts >2013-03-11 16:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [1cea420a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-74) [5186937e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-74) [5186937e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [5186937e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4db849e0 >2013-03-11 16:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-74) [5186937e] FINISH, SetVdsStatusVDSCommand, log id: 4db849e0 >2013-03-11 16:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-74) [5186937e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1786ae82 >2013-03-11 16:00:00,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [5186937e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1339a345 >2013-03-11 16:00:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-74) [5186937e] FINISH, GetHardwareInfoVDSCommand, log id: 1339a345 >2013-03-11 16:00:00,715 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-74) [6dc0433b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:00:00,718 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-74) [4025eadf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:00:00,719 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-74) [4025eadf] FINISH, ActivateVdsVDSCommand, log id: 1786ae82 >2013-03-11 16:00:00,743 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-74) [4025eadf] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:00:00,744 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-74) [4025eadf] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 16:00:00,746 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [4025eadf] Autorecovering 1 storage domains >2013-03-11 16:00:00,746 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-74) [4025eadf] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 16:00:00,747 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-74) [c29b170] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:00:00,747 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-74) [c29b170] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:00 PM >2013-03-11 16:00:00,753 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-74) [c29b170] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:00 PM >2013-03-11 16:00:00,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-85) [183a8084] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3b5284f4 >2013-03-11 16:00:00,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-85) [183a8084] FINISH, GetHardwareInfoVDSCommand, log id: 3b5284f4 >2013-03-11 16:00:00,819 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-85) [44ad63ce] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 16:00:00,822 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-85) [309052a4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 16:00:00,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4684c66 >2013-03-11 16:00:05,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 16:00:05,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:00:05,857 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 16:00:05,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 4684c66 >2013-03-11 16:00:05,861 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [7fc0f6cd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:00:05,884 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [7fc0f6cd] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2af6c481 >2013-03-11 16:00:05,884 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [7fc0f6cd] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 16:00:05,884 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [7fc0f6cd] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2af6c481 >2013-03-11 16:00:05,886 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [7fc0f6cd] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 16:00:05,887 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [309052a4] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 16:00:05,899 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-85) [23b74367] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:00:05,900 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-85) [23b74367] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7e25c567 >2013-03-11 16:00:05,902 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-85) [23b74367] FINISH, SetVdsStatusVDSCommand, log id: 7e25c567 >2013-03-11 16:00:05,907 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-85) [23b74367] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 16:00:05,931 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-85) [77e2c27b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:00:05,934 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-85) [24ac43db] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:00:05,935 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-85) [24ac43db] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 16:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) [645e4926] Autorecovering 1 hosts >2013-03-11 16:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) [645e4926] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-6) [4f892955] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-6) [4f892955] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-6) [4f892955] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 157f4e1e >2013-03-11 16:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-6) [4f892955] FINISH, SetVdsStatusVDSCommand, log id: 157f4e1e >2013-03-11 16:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-6) [4f892955] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 43803661 >2013-03-11 16:05:00,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-6) [4f892955] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 71e97b7f >2013-03-11 16:05:00,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-6) [4f892955] FINISH, GetHardwareInfoVDSCommand, log id: 71e97b7f >2013-03-11 16:05:00,097 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-6) [11f01f04] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:05:00,099 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-6) [7bf6e8a8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:05:00,101 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-6) [7bf6e8a8] FINISH, ActivateVdsVDSCommand, log id: 43803661 >2013-03-11 16:05:00,120 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-6) [7bf6e8a8] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:05:00,121 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-6) [7bf6e8a8] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 16:05:00,122 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) [7bf6e8a8] Autorecovering 1 storage domains >2013-03-11 16:05:00,122 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-6) [7bf6e8a8] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 16:05:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-6) [37c0b181] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:05:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-6) [37c0b181] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:05 PM >2013-03-11 16:05:00,128 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-6) [37c0b181] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:05 PM >2013-03-11 16:05:00,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [13b5f2cd] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3f2ec772 >2013-03-11 16:05:00,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-44) [13b5f2cd] FINISH, GetHardwareInfoVDSCommand, log id: 3f2ec772 >2013-03-11 16:05:00,231 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-44) [623be110] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 16:05:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-44) [3b7214f7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 16:05:00,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 647e371e >2013-03-11 16:05:04,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 16:05:04,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:05:04,225 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 16:05:04,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 647e371e >2013-03-11 16:05:04,232 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [4a3bc04a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:05:04,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [4a3bc04a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 49978218 >2013-03-11 16:05:04,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [4a3bc04a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 16:05:04,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [4a3bc04a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 49978218 >2013-03-11 16:05:04,264 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [4a3bc04a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 16:05:04,264 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [3b7214f7] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 16:05:04,292 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-44) [122b872f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:05:04,294 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [122b872f] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: ed5fd75 >2013-03-11 16:05:04,299 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-44) [122b872f] FINISH, SetVdsStatusVDSCommand, log id: ed5fd75 >2013-03-11 16:05:04,313 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-44) [122b872f] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 16:05:04,344 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-44) [9f9d176] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:05:04,346 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-44) [131033c6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:05:04,348 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-44) [131033c6] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 16:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [18412d77] Autorecovering 1 hosts >2013-03-11 16:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [18412d77] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [6f3ab698] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [6f3ab698] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [6f3ab698] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: ba3a0bf >2013-03-11 16:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [6f3ab698] FINISH, SetVdsStatusVDSCommand, log id: ba3a0bf >2013-03-11 16:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-22) [6f3ab698] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1e07ed33 >2013-03-11 16:10:00,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [6f3ab698] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 33362d14 >2013-03-11 16:10:00,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [6f3ab698] FINISH, GetHardwareInfoVDSCommand, log id: 33362d14 >2013-03-11 16:10:00,593 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [3b0aeb1e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:10:00,596 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [78d7075f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:10:00,597 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-22) [78d7075f] FINISH, ActivateVdsVDSCommand, log id: 1e07ed33 >2013-03-11 16:10:00,617 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [78d7075f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:10:00,618 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [78d7075f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 16:10:00,619 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [78d7075f] Autorecovering 1 storage domains >2013-03-11 16:10:00,619 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [78d7075f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 16:10:00,620 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [470e8b1c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:10:00,620 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [470e8b1c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:10 PM >2013-03-11 16:10:00,623 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [470e8b1c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:10 PM >2013-03-11 16:10:01,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [237fa769] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 26cd5d0c >2013-03-11 16:10:01,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [237fa769] FINISH, GetHardwareInfoVDSCommand, log id: 26cd5d0c >2013-03-11 16:10:01,649 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-80) [36473eaa] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 16:10:01,653 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-80) [77601f14] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 16:10:01,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2dca54cb >2013-03-11 16:10:07,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 16:10:07,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:10:07,656 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 16:10:07,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2dca54cb >2013-03-11 16:10:07,660 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [6ea590e8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:10:07,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6ea590e8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6e0766b8 >2013-03-11 16:10:07,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [6ea590e8] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 16:10:07,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [6ea590e8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6e0766b8 >2013-03-11 16:10:07,679 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [77601f14] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 16:10:07,698 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-80) [7822f209] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:10:07,700 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [7822f209] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7fdde54d >2013-03-11 16:10:07,701 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-80) [7822f209] FINISH, SetVdsStatusVDSCommand, log id: 7fdde54d >2013-03-11 16:10:07,707 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-80) [7822f209] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 16:10:07,731 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-80) [6df9630f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:10:07,733 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-80) [eb68cea] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:10:07,735 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-80) [eb68cea] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 16:15:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [2431912f] Autorecovering 1 hosts >2013-03-11 16:15:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [2431912f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:15:00,010 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [79ee7a52] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:15:00,011 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [79ee7a52] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:15:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [79ee7a52] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 64bb8af9 >2013-03-11 16:15:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-63) [79ee7a52] FINISH, SetVdsStatusVDSCommand, log id: 64bb8af9 >2013-03-11 16:15:00,019 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [79ee7a52] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 649fb4aa >2013-03-11 16:15:00,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [79ee7a52] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 471e9d68 >2013-03-11 16:15:00,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-63) [79ee7a52] FINISH, GetHardwareInfoVDSCommand, log id: 471e9d68 >2013-03-11 16:15:00,104 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-63) [6268529c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:15:00,107 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-63) [63bace15] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:15:00,108 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-63) [63bace15] FINISH, ActivateVdsVDSCommand, log id: 649fb4aa >2013-03-11 16:15:00,128 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-63) [63bace15] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:15:00,128 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-63) [63bace15] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 16:15:00,130 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [63bace15] Autorecovering 1 storage domains >2013-03-11 16:15:00,130 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-63) [63bace15] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 16:15:00,131 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [39b7c0bc] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:15:00,131 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [39b7c0bc] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:15 PM >2013-03-11 16:15:00,134 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-63) [39b7c0bc] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:15 PM >2013-03-11 16:15:01,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [3ea48131] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3c829b58 >2013-03-11 16:15:01,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-64) [3ea48131] FINISH, GetHardwareInfoVDSCommand, log id: 3c829b58 >2013-03-11 16:15:02,031 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-64) [1b523c87] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 16:15:02,035 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-64) [44715da9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 16:15:02,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 54609892 >2013-03-11 16:15:09,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 16:15:09,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:15:09,052 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 16:15:09,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 54609892 >2013-03-11 16:15:09,056 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [197edd80] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:15:09,070 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [197edd80] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6cd6e8f2 >2013-03-11 16:15:09,070 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [197edd80] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 16:15:09,070 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [197edd80] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6cd6e8f2 >2013-03-11 16:15:09,072 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [197edd80] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 16:15:09,072 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [44715da9] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 16:15:09,082 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-64) [7517963e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:15:09,084 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [7517963e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2afa5f7d >2013-03-11 16:15:09,085 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-64) [7517963e] FINISH, SetVdsStatusVDSCommand, log id: 2afa5f7d >2013-03-11 16:15:09,091 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-64) [7517963e] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 16:15:09,116 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-64) [5555d1ca] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:15:09,118 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-64) [24d55122] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:15:09,120 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-64) [24d55122] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 16:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [67ed1864] Autorecovering 1 hosts >2013-03-11 16:20:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [67ed1864] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-39) [7ed46b70] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-39) [7ed46b70] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:20:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [7ed46b70] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4b90a3b3 >2013-03-11 16:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [7ed46b70] FINISH, SetVdsStatusVDSCommand, log id: 4b90a3b3 >2013-03-11 16:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-39) [7ed46b70] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1fa6b79d >2013-03-11 16:20:00,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [7ed46b70] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 71ee4a0c >2013-03-11 16:20:00,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [7ed46b70] FINISH, GetHardwareInfoVDSCommand, log id: 71ee4a0c >2013-03-11 16:20:00,097 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-39) [bb84b9f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:20:00,099 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-39) [9e463e7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:20:00,101 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-39) [9e463e7] FINISH, ActivateVdsVDSCommand, log id: 1fa6b79d >2013-03-11 16:20:00,120 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-39) [9e463e7] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:20:00,121 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-39) [9e463e7] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 16:20:00,122 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [9e463e7] Autorecovering 1 storage domains >2013-03-11 16:20:00,122 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-39) [9e463e7] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 16:20:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-39) [390b6661] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:20:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-39) [390b6661] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:20 PM >2013-03-11 16:20:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-39) [390b6661] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:20 PM >2013-03-11 16:20:00,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [7176d9d0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 209e84ef >2013-03-11 16:20:00,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-83) [7176d9d0] FINISH, GetHardwareInfoVDSCommand, log id: 209e84ef >2013-03-11 16:20:00,407 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-83) [3f59b5eb] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 16:20:00,411 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-83) [186b4ad3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 16:20:00,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 40fcdeec >2013-03-11 16:20:06,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 16:20:06,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:20:06,431 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 16:20:06,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 40fcdeec >2013-03-11 16:20:06,435 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [39fb7ab2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:20:06,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [39fb7ab2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3391d6ce >2013-03-11 16:20:06,449 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [39fb7ab2] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 16:20:06,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [39fb7ab2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3391d6ce >2013-03-11 16:20:06,452 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [39fb7ab2] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 16:20:06,452 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [186b4ad3] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 16:20:06,462 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-83) [2adfd969] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:20:06,464 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [2adfd969] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 12e4d6c0 >2013-03-11 16:20:06,465 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-83) [2adfd969] FINISH, SetVdsStatusVDSCommand, log id: 12e4d6c0 >2013-03-11 16:20:06,471 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-83) [2adfd969] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 16:20:06,496 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-83) [3ac7a9d6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:20:06,499 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-83) [545dd468] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:20:06,500 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-83) [545dd468] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 16:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [23de5e2c] Autorecovering 1 hosts >2013-03-11 16:25:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [23de5e2c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-28) [4529cd6b] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-28) [4529cd6b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [4529cd6b] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 77f1ebc9 >2013-03-11 16:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [4529cd6b] FINISH, SetVdsStatusVDSCommand, log id: 77f1ebc9 >2013-03-11 16:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-28) [4529cd6b] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 683e826a >2013-03-11 16:25:00,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [4529cd6b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 69b6080d >2013-03-11 16:25:00,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [4529cd6b] FINISH, GetHardwareInfoVDSCommand, log id: 69b6080d >2013-03-11 16:25:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-28) [3a0132c6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:25:00,095 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-28) [71853e75] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:25:00,096 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-28) [71853e75] FINISH, ActivateVdsVDSCommand, log id: 683e826a >2013-03-11 16:25:00,116 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-28) [71853e75] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:25:00,117 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-28) [71853e75] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 16:25:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [71853e75] Autorecovering 1 storage domains >2013-03-11 16:25:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [71853e75] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 16:25:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-28) [38bd1c6e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:25:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-28) [38bd1c6e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:25 PM >2013-03-11 16:25:00,122 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-28) [38bd1c6e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:25 PM >2013-03-11 16:25:00,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [3d2e9d6d] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4595f519 >2013-03-11 16:25:00,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-53) [3d2e9d6d] FINISH, GetHardwareInfoVDSCommand, log id: 4595f519 >2013-03-11 16:25:00,820 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-53) [d75ba] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 16:25:00,824 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-53) [7105a7b0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 16:25:00,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 738a57bf >2013-03-11 16:25:07,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 16:25:07,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:25:07,822 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 16:25:07,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 738a57bf >2013-03-11 16:25:07,827 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [c0ee18b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:25:07,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [c0ee18b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2bdff3f3 >2013-03-11 16:25:07,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [c0ee18b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 16:25:07,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [c0ee18b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2bdff3f3 >2013-03-11 16:25:07,843 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [c0ee18b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 16:25:07,844 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [7105a7b0] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 16:25:07,854 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-53) [525a6eb5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:25:07,855 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [525a6eb5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 3268b957 >2013-03-11 16:25:07,856 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-53) [525a6eb5] FINISH, SetVdsStatusVDSCommand, log id: 3268b957 >2013-03-11 16:25:07,863 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-53) [525a6eb5] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 16:25:07,887 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-53) [5eec750e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:25:07,889 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-53) [39386dd9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:25:07,891 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-53) [39386dd9] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 16:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [664bb90] Autorecovering 1 hosts >2013-03-11 16:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [664bb90] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [a4e1c74] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [a4e1c74] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [a4e1c74] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: d71d860 >2013-03-11 16:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [a4e1c74] FINISH, SetVdsStatusVDSCommand, log id: d71d860 >2013-03-11 16:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-1) [a4e1c74] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 1c0ec805 >2013-03-11 16:30:00,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [a4e1c74] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 63610f83 >2013-03-11 16:30:00,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [a4e1c74] FINISH, GetHardwareInfoVDSCommand, log id: 63610f83 >2013-03-11 16:30:00,102 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-1) [4cde64d4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:30:00,104 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-1) [7bd328f2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:30:00,106 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-1) [7bd328f2] FINISH, ActivateVdsVDSCommand, log id: 1c0ec805 >2013-03-11 16:30:00,125 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-1) [7bd328f2] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:30:00,126 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-1) [7bd328f2] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 16:30:00,127 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [7bd328f2] Autorecovering 1 storage domains >2013-03-11 16:30:00,127 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-1) [7bd328f2] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 16:30:00,128 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [3506aa6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:30:00,128 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [3506aa6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:30 PM >2013-03-11 16:30:00,131 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-1) [3506aa6] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:30 PM >2013-03-11 16:30:02,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [63f7a017] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 12f5c869 >2013-03-11 16:30:02,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-87) [63f7a017] FINISH, GetHardwareInfoVDSCommand, log id: 12f5c869 >2013-03-11 16:30:02,188 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-87) [66636a94] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 16:30:02,192 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-87) [18e60f3c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 16:30:02,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6118bd76 >2013-03-11 16:30:08,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 16:30:08,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:30:08,244 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 16:30:08,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 6118bd76 >2013-03-11 16:30:08,248 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [5fc603d8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:30:08,262 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [5fc603d8] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5b16d064 >2013-03-11 16:30:08,262 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [5fc603d8] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 16:30:08,262 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [5fc603d8] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5b16d064 >2013-03-11 16:30:08,264 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [5fc603d8] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 16:30:08,264 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [18e60f3c] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 16:30:08,274 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-87) [666d42b0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:30:08,276 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [666d42b0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 696e4ec0 >2013-03-11 16:30:08,277 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-87) [666d42b0] FINISH, SetVdsStatusVDSCommand, log id: 696e4ec0 >2013-03-11 16:30:08,283 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-87) [666d42b0] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 16:30:08,307 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-87) [2cce7501] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:30:08,309 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-87) [54ee9186] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:30:08,311 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-87) [54ee9186] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 16:30:33,895 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-41) [3d4a3bd8] Start refreshing all users data >2013-03-11 16:30:35,103 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-56) [6ba4602a] Attempting to update VMs/Templates Ovf. >2013-03-11 16:30:35,104 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-56) [6ba4602a] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 16:30:35,112 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-56) [6ba4602a] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 16:30:35,112 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-56) [6ba4602a] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 16:30:35,115 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-56) [6ba4602a] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 16:30:35,115 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-56) [6ba4602a] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 16:30:35,116 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-56) [6ba4602a] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 16:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) [4c4f58bc] Autorecovering 1 hosts >2013-03-11 16:35:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) [4c4f58bc] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-30) [594dd161] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-30) [594dd161] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-30) [594dd161] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5ce3e098 >2013-03-11 16:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-30) [594dd161] FINISH, SetVdsStatusVDSCommand, log id: 5ce3e098 >2013-03-11 16:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-30) [594dd161] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: b6026cb >2013-03-11 16:35:00,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-30) [594dd161] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 14aea8f3 >2013-03-11 16:35:00,073 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-30) [594dd161] FINISH, GetHardwareInfoVDSCommand, log id: 14aea8f3 >2013-03-11 16:35:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-30) [7fcb7261] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:35:00,093 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-30) [655df0cb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:35:00,094 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-30) [655df0cb] FINISH, ActivateVdsVDSCommand, log id: b6026cb >2013-03-11 16:35:00,114 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-30) [655df0cb] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:35:00,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-30) [655df0cb] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 16:35:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) [655df0cb] Autorecovering 1 storage domains >2013-03-11 16:35:00,116 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-30) [655df0cb] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 16:35:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-30) [39899300] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:35:00,117 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-30) [39899300] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:35 PM >2013-03-11 16:35:00,120 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-30) [39899300] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:35 PM >2013-03-11 16:35:02,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [5e6e2788] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 60cfd077 >2013-03-11 16:35:02,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-48) [5e6e2788] FINISH, GetHardwareInfoVDSCommand, log id: 60cfd077 >2013-03-11 16:35:02,627 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-48) [1d19e89d] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 16:35:02,631 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-48) [5164264f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 16:35:02,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1437893f >2013-03-11 16:35:09,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 16:35:09,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:35:09,631 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 16:35:09,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 1437893f >2013-03-11 16:35:09,635 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [4353df30] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:35:09,650 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4353df30] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2daf59f8 >2013-03-11 16:35:09,650 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [4353df30] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 16:35:09,650 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [4353df30] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2daf59f8 >2013-03-11 16:35:09,652 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [4353df30] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 16:35:09,652 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [5164264f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 16:35:09,664 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-48) [63c14370] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:35:09,665 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [63c14370] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 321406cf >2013-03-11 16:35:09,666 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-48) [63c14370] FINISH, SetVdsStatusVDSCommand, log id: 321406cf >2013-03-11 16:35:09,673 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-48) [63c14370] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 16:35:09,699 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-48) [1bcc3ae8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:35:09,701 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-48) [8cda645] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:35:09,702 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-48) [8cda645] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 16:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [1f79cd8c] Autorecovering 1 hosts >2013-03-11 16:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [1f79cd8c] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:40:00,012 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [23c06fd5] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:40:00,012 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [23c06fd5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:40:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [23c06fd5] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 47e6b1b3 >2013-03-11 16:40:00,017 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-9) [23c06fd5] FINISH, SetVdsStatusVDSCommand, log id: 47e6b1b3 >2013-03-11 16:40:00,019 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-9) [23c06fd5] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4c95c52e >2013-03-11 16:40:00,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [23c06fd5] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4659874d >2013-03-11 16:40:00,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-9) [23c06fd5] FINISH, GetHardwareInfoVDSCommand, log id: 4659874d >2013-03-11 16:40:00,099 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-9) [69edd5ca] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:40:00,102 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-9) [42c30395] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:40:00,103 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-9) [42c30395] FINISH, ActivateVdsVDSCommand, log id: 4c95c52e >2013-03-11 16:40:00,123 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-9) [42c30395] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:40:00,124 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-9) [42c30395] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 16:40:00,125 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [42c30395] Autorecovering 1 storage domains >2013-03-11 16:40:00,125 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-9) [42c30395] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 16:40:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [74766f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:40:00,126 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [74766f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:40 PM >2013-03-11 16:40:00,129 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-9) [74766f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:40 PM >2013-03-11 16:40:00,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [342f8a71] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 584c1f5a >2013-03-11 16:40:00,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-10) [342f8a71] FINISH, GetHardwareInfoVDSCommand, log id: 584c1f5a >2013-03-11 16:40:00,993 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-10) [3bcc806c] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 16:40:00,997 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-10) [5dd61184] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 16:40:01,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 35ab19c2 >2013-03-11 16:40:08,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 16:40:08,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:40:08,004 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 16:40:08,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 35ab19c2 >2013-03-11 16:40:08,008 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [234b3956] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:40:08,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [234b3956] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 10f2dd8c >2013-03-11 16:40:08,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [234b3956] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 16:40:08,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [234b3956] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 10f2dd8c >2013-03-11 16:40:08,028 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [234b3956] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 16:40:08,028 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [5dd61184] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 16:40:08,043 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-10) [770ad213] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:40:08,044 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [770ad213] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 5c510b0 >2013-03-11 16:40:08,046 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-10) [770ad213] FINISH, SetVdsStatusVDSCommand, log id: 5c510b0 >2013-03-11 16:40:08,066 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-10) [770ad213] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 16:40:08,099 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-10) [273358cc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:40:08,102 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-10) [7b712e38] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:40:08,103 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-10) [7b712e38] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 16:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [1eec12ea] Autorecovering 1 hosts >2013-03-11 16:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [1eec12ea] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [61e6b8a7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [61e6b8a7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [61e6b8a7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7cff38bf >2013-03-11 16:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-88) [61e6b8a7] FINISH, SetVdsStatusVDSCommand, log id: 7cff38bf >2013-03-11 16:45:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-88) [61e6b8a7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3022c469 >2013-03-11 16:45:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [61e6b8a7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 58839eae >2013-03-11 16:45:00,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-88) [61e6b8a7] FINISH, GetHardwareInfoVDSCommand, log id: 58839eae >2013-03-11 16:45:00,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-88) [3f85cfb1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:45:00,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-88) [231e838b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:45:00,203 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-88) [231e838b] FINISH, ActivateVdsVDSCommand, log id: 3022c469 >2013-03-11 16:45:00,222 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-88) [231e838b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:45:00,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-88) [231e838b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 16:45:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [231e838b] Autorecovering 1 storage domains >2013-03-11 16:45:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-88) [231e838b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 16:45:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [46dff337] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:45:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [46dff337] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:45 PM >2013-03-11 16:45:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-88) [46dff337] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:45 PM >2013-03-11 16:45:02,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [407f3f2b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4b816409 >2013-03-11 16:45:02,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-97) [407f3f2b] FINISH, GetHardwareInfoVDSCommand, log id: 4b816409 >2013-03-11 16:45:02,399 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-97) [74e1da1d] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 16:45:02,403 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-97) [2924356e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 16:45:02,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1ae030f6 >2013-03-11 16:45:07,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 16:45:07,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:45:07,399 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 16:45:07,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 1ae030f6 >2013-03-11 16:45:07,403 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [db63239] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:45:07,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [db63239] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3749bb3a >2013-03-11 16:45:07,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [db63239] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 16:45:07,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [db63239] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3749bb3a >2013-03-11 16:45:07,419 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [db63239] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 16:45:07,420 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [2924356e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 16:45:07,430 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-97) [65679f0c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:45:07,431 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [65679f0c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 2ea253b1 >2013-03-11 16:45:07,432 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-97) [65679f0c] FINISH, SetVdsStatusVDSCommand, log id: 2ea253b1 >2013-03-11 16:45:07,438 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-97) [65679f0c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 16:45:07,465 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-97) [f56140f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:45:07,467 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-97) [44516e24] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:45:07,469 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-97) [44516e24] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 16:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [29f44e07] Autorecovering 1 hosts >2013-03-11 16:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [29f44e07] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-60) [3771e55e] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-60) [3771e55e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [3771e55e] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 14f9ccf7 >2013-03-11 16:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-60) [3771e55e] FINISH, SetVdsStatusVDSCommand, log id: 14f9ccf7 >2013-03-11 16:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-60) [3771e55e] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 4e384beb >2013-03-11 16:50:00,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [3771e55e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 62eb8d47 >2013-03-11 16:50:00,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-60) [3771e55e] FINISH, GetHardwareInfoVDSCommand, log id: 62eb8d47 >2013-03-11 16:50:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-60) [3aac5398] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:50:00,207 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-60) [238d1c3e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:50:00,208 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-60) [238d1c3e] FINISH, ActivateVdsVDSCommand, log id: 4e384beb >2013-03-11 16:50:00,228 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-60) [238d1c3e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:50:00,229 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-60) [238d1c3e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 16:50:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [238d1c3e] Autorecovering 1 storage domains >2013-03-11 16:50:00,230 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-60) [238d1c3e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 16:50:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-60) [1cdc98dd] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:50:00,231 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-60) [1cdc98dd] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:50 PM >2013-03-11 16:50:00,234 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-60) [1cdc98dd] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:50 PM >2013-03-11 16:50:01,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [54bac27e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1e964ebb >2013-03-11 16:50:01,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-11) [54bac27e] FINISH, GetHardwareInfoVDSCommand, log id: 1e964ebb >2013-03-11 16:50:01,756 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-11) [c8344ef] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 16:50:01,759 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-11) [b45ebcc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 16:50:01,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 82a10dd >2013-03-11 16:50:07,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 16:50:07,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:50:07,756 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 16:50:07,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 82a10dd >2013-03-11 16:50:07,760 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [3eaaf7a7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:50:07,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [3eaaf7a7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7ac5d7d >2013-03-11 16:50:07,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [3eaaf7a7] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 16:50:07,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [3eaaf7a7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7ac5d7d >2013-03-11 16:50:07,776 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [3eaaf7a7] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 16:50:07,776 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [b45ebcc] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 16:50:07,786 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-11) [410d4ef8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:50:07,787 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [410d4ef8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 686f41c1 >2013-03-11 16:50:07,789 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-11) [410d4ef8] FINISH, SetVdsStatusVDSCommand, log id: 686f41c1 >2013-03-11 16:50:07,794 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-11) [410d4ef8] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 16:50:07,828 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-11) [6016611] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:50:07,831 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-11) [291c51f9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:50:07,832 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-11) [291c51f9] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 16:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [6b6d9112] Autorecovering 1 hosts >2013-03-11 16:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [6b6d9112] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-15) [3cefcf29] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-15) [3cefcf29] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [3cefcf29] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 79699544 >2013-03-11 16:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [3cefcf29] FINISH, SetVdsStatusVDSCommand, log id: 79699544 >2013-03-11 16:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-15) [3cefcf29] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6f1ded85 >2013-03-11 16:55:00,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [3cefcf29] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 27b9ffe7 >2013-03-11 16:55:00,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [3cefcf29] FINISH, GetHardwareInfoVDSCommand, log id: 27b9ffe7 >2013-03-11 16:55:00,412 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-15) [6214bad7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:55:00,415 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-15) [3f031bfc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:55:00,416 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-15) [3f031bfc] FINISH, ActivateVdsVDSCommand, log id: 6f1ded85 >2013-03-11 16:55:00,436 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-15) [3f031bfc] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 16:55:00,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-15) [3f031bfc] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 16:55:00,438 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [3f031bfc] Autorecovering 1 storage domains >2013-03-11 16:55:00,438 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-15) [3f031bfc] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 16:55:00,439 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-15) [279dc86b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:55:00,439 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-15) [279dc86b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 4:55 PM >2013-03-11 16:55:00,442 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-15) [279dc86b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 4:55 PM >2013-03-11 16:55:02,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [38bd1c6e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 58e83373 >2013-03-11 16:55:02,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [38bd1c6e] FINISH, GetHardwareInfoVDSCommand, log id: 58e83373 >2013-03-11 16:55:02,126 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-28) [5e48caee] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 16:55:02,130 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-28) [1950cb6f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 16:55:02,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 43bc0b26 >2013-03-11 16:55:08,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 16:55:08,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 16:55:08,114 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 16:55:08,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 43bc0b26 >2013-03-11 16:55:08,118 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [5948929e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 16:55:08,133 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5948929e] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6412f0fc >2013-03-11 16:55:08,133 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [5948929e] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 16:55:08,133 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5948929e] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6412f0fc >2013-03-11 16:55:08,136 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [5948929e] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 16:55:08,136 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [1950cb6f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 16:55:08,146 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-28) [75d55595] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:55:08,148 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [75d55595] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 27e82068 >2013-03-11 16:55:08,149 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [75d55595] FINISH, SetVdsStatusVDSCommand, log id: 27e82068 >2013-03-11 16:55:08,155 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-28) [75d55595] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 16:55:08,180 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-28) [7463706f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:55:08,182 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-28) [7845ae28] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 16:55:08,184 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-28) [7845ae28] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 17:00:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [9fe5421] Autorecovering 1 hosts >2013-03-11 17:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [9fe5421] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-91) [728f0c3c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-91) [728f0c3c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:00:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [728f0c3c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4bbac59e >2013-03-11 17:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-91) [728f0c3c] FINISH, SetVdsStatusVDSCommand, log id: 4bbac59e >2013-03-11 17:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-91) [728f0c3c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 62e4ee17 >2013-03-11 17:00:00,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [728f0c3c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5ef4a690 >2013-03-11 17:00:00,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-91) [728f0c3c] FINISH, GetHardwareInfoVDSCommand, log id: 5ef4a690 >2013-03-11 17:00:00,212 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-91) [18498a3a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:00:00,215 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-91) [74673ab4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:00:00,216 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-91) [74673ab4] FINISH, ActivateVdsVDSCommand, log id: 62e4ee17 >2013-03-11 17:00:00,235 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-91) [74673ab4] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:00:00,236 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-91) [74673ab4] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 17:00:00,238 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [74673ab4] Autorecovering 1 storage domains >2013-03-11 17:00:00,238 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-91) [74673ab4] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 17:00:00,238 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-91) [7c2b46bf] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:00:00,238 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-91) [7c2b46bf] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:00 PM >2013-03-11 17:00:00,241 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-91) [7c2b46bf] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:00 PM >2013-03-11 17:00:02,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [54b6b7fc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3baf98a3 >2013-03-11 17:00:02,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-45) [54b6b7fc] FINISH, GetHardwareInfoVDSCommand, log id: 3baf98a3 >2013-03-11 17:00:02,513 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-45) [3d9c4aa4] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 17:00:02,516 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-45) [1a93e200] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 17:00:02,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6ceda667 >2013-03-11 17:00:08,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 17:00:08,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:00:08,567 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 17:00:08,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 6ceda667 >2013-03-11 17:00:08,571 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [58bb5361] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:00:08,585 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [58bb5361] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 597488b7 >2013-03-11 17:00:08,585 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [58bb5361] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 17:00:08,585 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [58bb5361] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 597488b7 >2013-03-11 17:00:08,587 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [58bb5361] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 17:00:08,588 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [1a93e200] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 17:00:08,598 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-45) [679ea695] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:00:08,599 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [679ea695] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 77562138 >2013-03-11 17:00:08,600 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-45) [679ea695] FINISH, SetVdsStatusVDSCommand, log id: 77562138 >2013-03-11 17:00:08,606 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-45) [679ea695] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 17:00:08,631 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-45) [2f66b2de] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:00:08,633 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-45) [24cb0e25] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:00:08,635 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-45) [24cb0e25] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 17:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [919b576] Autorecovering 1 hosts >2013-03-11 17:05:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [919b576] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-4) [3698edd8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-4) [3698edd8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [3698edd8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3e148855 >2013-03-11 17:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-4) [3698edd8] FINISH, SetVdsStatusVDSCommand, log id: 3e148855 >2013-03-11 17:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-4) [3698edd8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 2cef59a1 >2013-03-11 17:05:00,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [3698edd8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3fb72e8d >2013-03-11 17:05:00,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-4) [3698edd8] FINISH, GetHardwareInfoVDSCommand, log id: 3fb72e8d >2013-03-11 17:05:00,441 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-4) [9524cef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:05:00,444 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-4) [71adb563] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:05:00,445 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-4) [71adb563] FINISH, ActivateVdsVDSCommand, log id: 2cef59a1 >2013-03-11 17:05:00,465 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-4) [71adb563] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:05:00,466 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-4) [71adb563] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 17:05:00,467 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [71adb563] Autorecovering 1 storage domains >2013-03-11 17:05:00,467 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-4) [71adb563] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 17:05:00,468 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-4) [527fb68b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:05:00,468 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-4) [527fb68b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:05 PM >2013-03-11 17:05:00,471 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-4) [527fb68b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:05 PM >2013-03-11 17:05:02,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [390b6661] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 693ddc19 >2013-03-11 17:05:02,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-39) [390b6661] FINISH, GetHardwareInfoVDSCommand, log id: 693ddc19 >2013-03-11 17:05:02,925 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-39) [760ef181] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 17:05:02,929 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-39) [6add2c5a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 17:05:02,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6bedda5c >2013-03-11 17:05:07,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 17:05:07,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:05:07,947 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 17:05:07,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 6bedda5c >2013-03-11 17:05:07,951 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [110fd924] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:05:07,964 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [110fd924] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2e0f86a9 >2013-03-11 17:05:07,964 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [110fd924] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 17:05:07,964 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [110fd924] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2e0f86a9 >2013-03-11 17:05:07,967 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [110fd924] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 17:05:07,967 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [6add2c5a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 17:05:07,977 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-39) [44f7a11d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:05:07,978 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [44f7a11d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 19ba5211 >2013-03-11 17:05:07,979 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-39) [44f7a11d] FINISH, SetVdsStatusVDSCommand, log id: 19ba5211 >2013-03-11 17:05:07,985 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-39) [44f7a11d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 17:05:08,010 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-39) [49a8c708] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:05:08,012 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-39) [50fe5b84] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:05:08,014 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-39) [50fe5b84] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 17:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [11bfc989] Autorecovering 1 hosts >2013-03-11 17:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [11bfc989] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [2f6cb57] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [2f6cb57] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [2f6cb57] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3fd34209 >2013-03-11 17:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-77) [2f6cb57] FINISH, SetVdsStatusVDSCommand, log id: 3fd34209 >2013-03-11 17:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-77) [2f6cb57] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 74cb27f7 >2013-03-11 17:10:00,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [2f6cb57] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5a421115 >2013-03-11 17:10:00,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-77) [2f6cb57] FINISH, GetHardwareInfoVDSCommand, log id: 5a421115 >2013-03-11 17:10:00,092 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-77) [4ebafb9f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:10:00,095 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-77) [46f20b45] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:10:00,096 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-77) [46f20b45] FINISH, ActivateVdsVDSCommand, log id: 74cb27f7 >2013-03-11 17:10:00,115 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-77) [46f20b45] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:10:00,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-77) [46f20b45] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 17:10:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [46f20b45] Autorecovering 1 storage domains >2013-03-11 17:10:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-77) [46f20b45] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 17:10:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [cf61334] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:10:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [cf61334] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:10 PM >2013-03-11 17:10:00,121 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-77) [cf61334] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:10 PM >2013-03-11 17:10:02,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [279dc86b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 49b4894d >2013-03-11 17:10:02,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [279dc86b] FINISH, GetHardwareInfoVDSCommand, log id: 49b4894d >2013-03-11 17:10:02,303 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-15) [1d9abd72] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 17:10:02,306 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-15) [45af4873] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 17:10:02,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 559c57ca >2013-03-11 17:10:07,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 17:10:07,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:10:07,306 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 17:10:07,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 559c57ca >2013-03-11 17:10:07,311 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [e7f2317] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:10:07,324 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [e7f2317] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 5ace7f62 >2013-03-11 17:10:07,324 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [e7f2317] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 17:10:07,324 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [e7f2317] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 5ace7f62 >2013-03-11 17:10:07,326 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [e7f2317] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 17:10:07,327 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [45af4873] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 17:10:07,340 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-15) [64370eca] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:10:07,341 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [64370eca] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 168fc2bc >2013-03-11 17:10:07,342 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-15) [64370eca] FINISH, SetVdsStatusVDSCommand, log id: 168fc2bc >2013-03-11 17:10:07,348 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-15) [64370eca] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 17:10:07,381 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-15) [6017f339] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:10:07,383 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-15) [2e714cfa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:10:07,385 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-15) [2e714cfa] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 17:12:19,046 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginAdminUserCommand internal: false. >2013-03-11 17:12:19,127 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 17:12:19,133 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:12:19,133 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:12:20,845 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (WebAdminUpperRightButtonURL) with null version, using default general for version >2013-03-11 17:12:36,729 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-3) [6f70592f] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-11 17:12:53,671 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-10) [1e714b11] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: da9b3f49-2f13-482b-9b6a-55da6dd7dbfb Type: StoragePool >2013-03-11 17:13:10,906 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-5) [4aab77ea] Running command: AddVdsCommand internal: false. Entities affected : ID: 6a3926cd-1ef9-4303-8cb9-0842e0ee44e7 Type: VdsGroups >2013-03-11 17:13:10,940 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-5) [244fe6fc] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 3b71db53-3a3a-4daa-a66d-83c2dadad669 Type: VDS >2013-03-11 17:13:10,944 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-5) [244fe6fc] Lock freed to object EngineLock [exclusiveLocks= key: da9b3f49-2f13-482b-9b6a-55da6dd7dbfb value: REGISTER_VDS >, sharedLocks= ] >2013-03-11 17:13:10,945 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-5) [244fe6fc] Cannot get vdsManager for vdsid=3b71db53-3a3a-4daa-a66d-83c2dadad669 >2013-03-11 17:13:10,949 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [244fe6fc] START, RemoveVdsVDSCommand(HostName = purple-vds1.qa.lab.tlv.redhat.com, HostId = 3b71db53-3a3a-4daa-a66d-83c2dadad669), log id: 6a26d388 >2013-03-11 17:13:10,949 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-5) [244fe6fc] Cannot get vdsManager for vdsid=3b71db53-3a3a-4daa-a66d-83c2dadad669 >2013-03-11 17:13:10,949 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [244fe6fc] FINISH, RemoveVdsVDSCommand, log id: 6a26d388 >2013-03-11 17:13:10,950 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-5) [244fe6fc] Cannot get vdsManager for vdsid=3b71db53-3a3a-4daa-a66d-83c2dadad669 >2013-03-11 17:13:10,951 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [244fe6fc] START, AddVdsVDSCommand(HostName = purple-vds1.qa.lab.tlv.redhat.com, HostId = 3b71db53-3a3a-4daa-a66d-83c2dadad669), log id: 7c596577 >2013-03-11 17:13:10,951 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [244fe6fc] AddVds - entered , starting logic to add VDS 3b71db53-3a3a-4daa-a66d-83c2dadad669 >2013-03-11 17:13:10,952 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [244fe6fc] AddVds - VDS 3b71db53-3a3a-4daa-a66d-83c2dadad669 was added, will try to add it to the resource manager >2013-03-11 17:13:10,953 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-5) [244fe6fc] Eneterd VdsManager:constructor >2013-03-11 17:13:10,953 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-5) [244fe6fc] vdsBroker(purple-vds1.qa.lab.tlv.redhat.com,54,321) >2013-03-11 17:13:10,954 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-5) [244fe6fc] ResourceManager::AddVds - VDS 3b71db53-3a3a-4daa-a66d-83c2dadad669 was added to the Resource Manager >2013-03-11 17:13:10,954 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [244fe6fc] FINISH, AddVdsVDSCommand, log id: 7c596577 >2013-03-11 17:13:10,989 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-46) [6596a555] Running command: InstallVdsCommand internal: true. Entities affected : ID: 3b71db53-3a3a-4daa-a66d-83c2dadad669 Type: VDS >2013-03-11 17:13:10,989 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-46) [6596a555] Before Installation host 3b71db53-3a3a-4daa-a66d-83c2dadad669, purple-vds1.qa.lab.tlv.redhat.com >2013-03-11 17:13:11,102 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-3-thread-46) [6596a555] Installation purple-vds1.qa.lab.tlv.redhat.com: Connected to host purple-vds1.qa.lab.tlv.redhat.com with SSH key fingerprint: 13:12:03:93:ea:33:e7:e8:ae:98:fe:a3:f3:82:ea:08 >2013-03-11 17:13:11,139 INFO [org.ovirt.engine.core.bll.VdsDeploy] (pool-3-thread-46) [6596a555] Installation of purple-vds1.qa.lab.tlv.redhat.com. Executing command via SSH umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2013-03-11 17:13:11,202 INFO [org.ovirt.engine.core.utils.ssh.SSHDialog] (pool-3-thread-46) SSH execute root@purple-vds1.qa.lab.tlv.redhat.com 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2013-03-11 17:13:12,071 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Stage: Initializing >2013-03-11 17:13:12,091 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Stage: Environment setup >2013-03-11 17:13:12,102 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Stage: Installation packages setup >2013-03-11 17:13:12,108 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Stage: Programs detection >2013-03-11 17:13:12,113 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Stage: Environment customization >2013-03-11 17:13:12,128 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Host is ovirt-node >2013-03-11 17:13:12,137 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Logs at host located at: '/tmp/ovirt-host-deploy-20130311151357.log' >2013-03-11 17:13:12,174 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host purple-vds1.qa.lab.tlv.redhat.com reports unique id 9978F05A-B189-11DE-9BD8-00215EC7F8AC >2013-03-11 17:13:12,193 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 9978F05A-B189-11DE-9BD8-00215EC7F8AC to Host purple-vds1.qa.lab.tlv.redhat.com >2013-03-11 17:13:12,239 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Stage: Setup validation >2013-03-11 17:13:12,299 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Hardware supports virtualization >2013-03-11 17:13:12,304 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Stage: Transaction setup >2013-03-11 17:13:12,311 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Stage: Package installation >2013-03-11 17:13:13,733 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Stage: Misc configuration >2013-03-11 17:13:13,772 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Setting time >2013-03-11 17:13:13,956 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-6) [37c0b181] Initializing Host: purple-vds1.qa.lab.tlv.redhat.com >2013-03-11 17:13:15,491 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Enrolling certificate >2013-03-11 17:13:16,563 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Stage: Transaction commit >2013-03-11 17:13:16,570 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Stage: Closing up >2013-03-11 17:13:17,899 ERROR [org.ovirt.engine.core.bll.GetoVirtISOsQuery] (ajp-/127.0.0.1:8702-2) ovirt ISOs directory not found. Search in: /usr/share/ovirt-node-iso >2013-03-11 17:13:19,200 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-11 17:13:19,207 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:13:19,207 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:13:22,739 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Starting vdsm >2013-03-11 17:13:31,781 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Stage: Pre-termination >2013-03-11 17:13:31,798 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20130311171331-purple-vds1.qa.lab.tlv.redhat.com-6596a555.log' >2013-03-11 17:13:32,039 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation purple-vds1.qa.lab.tlv.redhat.com: Stage: Termination >2013-03-11 17:13:32,129 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-46) [6596a555] After Installation host purple-vds1.qa.lab.tlv.redhat.com, oVirtNode >2013-03-11 17:13:35,281 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-74) [7c5131f6] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 17:13:35,292 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-46) [7c5131f6] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2013-03-11 17:13:35,296 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-74) [219b3b1d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 3b71db53-3a3a-4daa-a66d-83c2dadad669 Type: VDS >2013-03-11 17:13:35,300 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-74) [4bee7c39] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 3b71db53-3a3a-4daa-a66d-83c2dadad669 Type: VDS >2013-03-11 17:14:19,198 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-11 17:14:19,207 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:14:19,208 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:14:51,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-7) [5b93d1e8] START, ValidateStorageServerConnectionVDSCommand(HostName = purple-vds1.qa.lab.tlv.redhat.com, HostId = 3b71db53-3a3a-4daa-a66d-83c2dadad669, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: wolf.qa.lab.tlv.redhat.com:/volumes/wolf/nexenta, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 37cfdd1f >2013-03-11 17:14:51,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-7) [5b93d1e8] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 37cfdd1f >2013-03-11 17:14:51,450 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-7) [5b93d1e8] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-11 17:14:51,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [5b93d1e8] START, ConnectStorageServerVDSCommand(HostName = purple-vds1.qa.lab.tlv.redhat.com, HostId = 3b71db53-3a3a-4daa-a66d-83c2dadad669, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: wolf.qa.lab.tlv.redhat.com:/volumes/wolf/nexenta, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5c44beef >2013-03-11 17:14:51,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [5b93d1e8] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 5c44beef >2013-03-11 17:14:51,871 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-6) [414a87fb] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-11 17:14:51,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-6) [414a87fb] START, CreateStorageDomainVDSCommand(HostName = purple-vds1.qa.lab.tlv.redhat.com, HostId = 3b71db53-3a3a-4daa-a66d-83c2dadad669, storageDomain=org.ovirt.engine.core.common.businessentities.StorageDomainStatic@3d7ae896, args=wolf.qa.lab.tlv.redhat.com:/volumes/wolf/nexenta), log id: e1233f5 >2013-03-11 17:14:57,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-6) [414a87fb] FINISH, CreateStorageDomainVDSCommand, log id: e1233f5 >2013-03-11 17:14:57,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-6) [414a87fb] START, GetStorageDomainStatsVDSCommand(HostName = purple-vds1.qa.lab.tlv.redhat.com, HostId = 3b71db53-3a3a-4daa-a66d-83c2dadad669, storageDomainId=4f8233bb-eb7c-4f6b-8b92-7ed198c666bf), log id: 2f80ac65 >2013-03-11 17:14:57,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-6) [414a87fb] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.storage_domains@4c1f3425, log id: 2f80ac65 >2013-03-11 17:14:57,510 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-9) [3c01479a] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-11 17:14:57,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [3c01479a] START, DisconnectStorageServerVDSCommand(HostName = purple-vds1.qa.lab.tlv.redhat.com, HostId = 3b71db53-3a3a-4daa-a66d-83c2dadad669, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: wolf.qa.lab.tlv.redhat.com:/volumes/wolf/nexenta, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 28f1d33 >2013-03-11 17:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [336806d] Autorecovering 1 hosts >2013-03-11 17:15:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [336806d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [218b1fbf] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [218b1fbf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [218b1fbf] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3a86494c >2013-03-11 17:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [218b1fbf] FINISH, SetVdsStatusVDSCommand, log id: 3a86494c >2013-03-11 17:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [218b1fbf] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 49d9c6c9 >2013-03-11 17:15:00,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [218b1fbf] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 23b9ef07 >2013-03-11 17:15:00,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [218b1fbf] FINISH, GetHardwareInfoVDSCommand, log id: 23b9ef07 >2013-03-11 17:15:00,205 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [5a100806] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:15:00,208 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [1aae4c6f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:15:00,209 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-59) [1aae4c6f] FINISH, ActivateVdsVDSCommand, log id: 49d9c6c9 >2013-03-11 17:15:00,229 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-59) [1aae4c6f] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:15:00,230 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [1aae4c6f] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 17:15:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [1aae4c6f] Autorecovering 1 storage domains >2013-03-11 17:15:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-59) [1aae4c6f] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 17:15:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [72b0e656] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:15:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [72b0e656] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:15 PM >2013-03-11 17:15:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-59) [72b0e656] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:15 PM >2013-03-11 17:15:01,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [470e8b1c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 46a94c67 >2013-03-11 17:15:01,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-9) [3c01479a] FINISH, DisconnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 28f1d33 >2013-03-11 17:15:01,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [470e8b1c] FINISH, GetHardwareInfoVDSCommand, log id: 46a94c67 >2013-03-11 17:15:01,705 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-10) [41edd5d9] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 4f8233bb-eb7c-4f6b-8b92-7ed198c666bf Type: Storage >2013-03-11 17:15:01,715 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-10) [14b2fc5b] Lock Acquired to object EngineLock [exclusiveLocks= key: da9b3f49-2f13-482b-9b6a-55da6dd7dbfb value: POOL >, sharedLocks= ] >2013-03-11 17:15:01,718 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-22) [f677f7c] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 17:15:01,723 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-22) [36c6e620] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 17:15:01,728 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-10) [14b2fc5b] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected : ID: da9b3f49-2f13-482b-9b6a-55da6dd7dbfb Type: StoragePool >2013-03-11 17:15:01,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5a87eb53 >2013-03-11 17:15:01,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-10) [5dffa0f6] START, ValidateStorageServerConnectionVDSCommand(HostName = purple-vds1.qa.lab.tlv.redhat.com, HostId = 3b71db53-3a3a-4daa-a66d-83c2dadad669, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 150a037f-ccee-480a-8ca9-dcfadfe7213d, connection: wolf.qa.lab.tlv.redhat.com:/volumes/wolf/nexenta, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 28d4863e >2013-03-11 17:15:01,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-10) [5dffa0f6] FINISH, ValidateStorageServerConnectionVDSCommand, return: {150a037f-ccee-480a-8ca9-dcfadfe7213d=0}, log id: 28d4863e >2013-03-11 17:15:01,810 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8702-10) [5dffa0f6] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-11 17:15:01,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [5dffa0f6] START, ConnectStorageServerVDSCommand(HostName = purple-vds1.qa.lab.tlv.redhat.com, HostId = 3b71db53-3a3a-4daa-a66d-83c2dadad669, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 150a037f-ccee-480a-8ca9-dcfadfe7213d, connection: wolf.qa.lab.tlv.redhat.com:/volumes/wolf/nexenta, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 64e11567 >2013-03-11 17:15:01,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [5dffa0f6] FINISH, ConnectStorageServerVDSCommand, return: {150a037f-ccee-480a-8ca9-dcfadfe7213d=0}, log id: 64e11567 >2013-03-11 17:15:01,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-10) [5dffa0f6] START, CreateStoragePoolVDSCommand(HostName = purple-vds1.qa.lab.tlv.redhat.com, HostId = 3b71db53-3a3a-4daa-a66d-83c2dadad669, storagePoolId=da9b3f49-2f13-482b-9b6a-55da6dd7dbfb, storageType=NFS, storagePoolName=rhevh1, masterDomainId=4f8233bb-eb7c-4f6b-8b92-7ed198c666bf, domainsIdList=[4f8233bb-eb7c-4f6b-8b92-7ed198c666bf], masterVersion=1), log id: 2630ecd1 >2013-03-11 17:15:03,135 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8702-10) [5dffa0f6] Weird return value: StatusForXmlRpc [mCode=661, mMessage=Cannot acquire host id: ('4f8233bb-eb7c-4f6b-8b92-7ed198c666bf', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))] >2013-03-11 17:15:03,135 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8702-10) [5dffa0f6] Weird return value: StatusForXmlRpc [mCode=661, mMessage=Cannot acquire host id: ('4f8233bb-eb7c-4f6b-8b92-7ed198c666bf', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))] >2013-03-11 17:15:03,135 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8702-10) [5dffa0f6] Failed in CreateStoragePoolVDS method >2013-03-11 17:15:03,136 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8702-10) [5dffa0f6] Error code unexpected and error message VDSGenericException: VDSErrorException: Failed to CreateStoragePoolVDS, error = Cannot acquire host id: ('4f8233bb-eb7c-4f6b-8b92-7ed198c666bf', SanlockException(19, 'Sanlock lockspace add failure', 'No such device')) >2013-03-11 17:15:03,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8702-10) [5dffa0f6] Command org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=661, mMessage=Cannot acquire host id: ('4f8233bb-eb7c-4f6b-8b92-7ed198c666bf', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))]] >2013-03-11 17:15:03,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8702-10) [5dffa0f6] HostName = purple-vds1.qa.lab.tlv.redhat.com >2013-03-11 17:15:03,136 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (ajp-/127.0.0.1:8702-10) [5dffa0f6] Command CreateStoragePoolVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStoragePoolVDS, error = Cannot acquire host id: ('4f8233bb-eb7c-4f6b-8b92-7ed198c666bf', SanlockException(19, 'Sanlock lockspace add failure', 'No such device')) >2013-03-11 17:15:03,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-10) [5dffa0f6] FINISH, CreateStoragePoolVDSCommand, log id: 2630ecd1 >2013-03-11 17:15:03,136 ERROR [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-10) [5dffa0f6] Command org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStoragePoolVDS, error = Cannot acquire host id: ('4f8233bb-eb7c-4f6b-8b92-7ed198c666bf', SanlockException(19, 'Sanlock lockspace add failure', 'No such device')) >2013-03-11 17:15:03,138 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-10) [5dffa0f6] Lock freed to object EngineLock [exclusiveLocks= key: da9b3f49-2f13-482b-9b6a-55da6dd7dbfb value: POOL >, sharedLocks= ] >2013-03-11 17:15:03,171 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-10) [5dffa0f6] Command [id=7a918687-8269-4f4c-9b79-a98d15e801d6]: Compensating CHANGED_ENTITY of org.ovirt.engine.core.common.businessentities.storage_pool; snapshot: id=da9b3f49-2f13-482b-9b6a-55da6dd7dbfb. >2013-03-11 17:15:03,180 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-10) [5dffa0f6] Command [id=7a918687-8269-4f4c-9b79-a98d15e801d6]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: storagePoolId = da9b3f49-2f13-482b-9b6a-55da6dd7dbfb, storageId = 4f8233bb-eb7c-4f6b-8b92-7ed198c666bf. >2013-03-11 17:15:03,192 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-10) [5dffa0f6] Command [id=7a918687-8269-4f4c-9b79-a98d15e801d6]: Compensating CHANGED_ENTITY of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: id=4f8233bb-eb7c-4f6b-8b92-7ed198c666bf. >2013-03-11 17:15:05,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 17:15:05,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:15:05,728 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 17:15:05,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 5a87eb53 >2013-03-11 17:15:05,732 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [c9019e6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:15:05,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [c9019e6] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 2bf94a92 >2013-03-11 17:15:05,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [c9019e6] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 17:15:05,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [c9019e6] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 2bf94a92 >2013-03-11 17:15:05,757 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [36c6e620] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 17:15:05,768 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-22) [73fc9050] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:15:05,770 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [73fc9050] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 12a6cd04 >2013-03-11 17:15:05,771 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [73fc9050] FINISH, SetVdsStatusVDSCommand, log id: 12a6cd04 >2013-03-11 17:15:05,778 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [73fc9050] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 17:15:05,804 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [64ba5bc8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:15:05,807 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [5ea767a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:15:05,808 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-22) [5ea767a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 17:15:19,197 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-11 17:15:19,205 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:15:19,205 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:16:19,200 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-11 17:16:19,212 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:16:19,212 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:17:19,202 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-11 17:17:19,209 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:17:19,209 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:18:19,201 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-11 17:18:19,209 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:18:19,209 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:19:19,207 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-11 17:19:19,216 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:19:19,216 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [58bc9d93] Autorecovering 1 hosts >2013-03-11 17:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [58bc9d93] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-38) [77577221] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:20:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-38) [77577221] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [77577221] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 42be161a >2013-03-11 17:20:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-38) [77577221] FINISH, SetVdsStatusVDSCommand, log id: 42be161a >2013-03-11 17:20:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-38) [77577221] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7da43809 >2013-03-11 17:20:00,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [77577221] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 9d86c95 >2013-03-11 17:20:00,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-38) [77577221] FINISH, GetHardwareInfoVDSCommand, log id: 9d86c95 >2013-03-11 17:20:00,540 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-38) [59529183] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:20:00,543 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-38) [160ed50] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:20:00,544 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-38) [160ed50] FINISH, ActivateVdsVDSCommand, log id: 7da43809 >2013-03-11 17:20:00,567 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-38) [160ed50] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:20:00,572 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [160ed50] Autorecovering 1 storage domains >2013-03-11 17:20:00,572 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-38) [160ed50] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 17:20:00,573 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-38) [18b6f504] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:20:00,573 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-38) [18b6f504] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:20 PM >2013-03-11 17:20:00,576 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-38) [18b6f504] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:20 PM >2013-03-11 17:20:03,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [3506aa6] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1fbc17ab >2013-03-11 17:20:03,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-1) [3506aa6] FINISH, GetHardwareInfoVDSCommand, log id: 1fbc17ab >2013-03-11 17:20:03,108 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-1) [4711424e] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 17:20:03,112 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-1) [6be09e84] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 17:20:03,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 66a147bd >2013-03-11 17:20:11,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 17:20:11,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:20:11,104 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 17:20:11,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 66a147bd >2013-03-11 17:20:11,108 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [3cc00e10] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:20:11,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [3cc00e10] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1dc41a16 >2013-03-11 17:20:11,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [3cc00e10] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 17:20:11,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [3cc00e10] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1dc41a16 >2013-03-11 17:20:11,125 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [3cc00e10] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 17:20:11,125 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [6be09e84] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 17:20:11,136 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-1) [186054ae] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:20:11,137 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [186054ae] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 53fd6c3e >2013-03-11 17:20:11,139 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-1) [186054ae] FINISH, SetVdsStatusVDSCommand, log id: 53fd6c3e >2013-03-11 17:20:11,146 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-1) [186054ae] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 17:20:11,172 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-1) [201e79a7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:20:11,175 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-1) [255510ac] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:20:11,176 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-1) [255510ac] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 17:20:19,204 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-11 17:20:19,212 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:20:19,212 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:21:19,206 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-11 17:21:19,213 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:21:19,213 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:22:19,232 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-11 17:22:19,239 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:22:19,240 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:23:19,221 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-11 17:23:19,232 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:23:19,232 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:24:19,214 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-11 17:24:19,223 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:24:19,223 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:25:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [5e282629] Autorecovering 1 hosts >2013-03-11 17:25:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [5e282629] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:25:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [5d7e2421] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:25:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [5d7e2421] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:25:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [5d7e2421] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2713500a >2013-03-11 17:25:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [5d7e2421] FINISH, SetVdsStatusVDSCommand, log id: 2713500a >2013-03-11 17:25:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [5d7e2421] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 47421a79 >2013-03-11 17:25:00,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [5d7e2421] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 236e42d1 >2013-03-11 17:25:00,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [5d7e2421] FINISH, GetHardwareInfoVDSCommand, log id: 236e42d1 >2013-03-11 17:25:00,096 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [1864222d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:25:00,099 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [7911aa45] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:25:00,101 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [7911aa45] FINISH, ActivateVdsVDSCommand, log id: 47421a79 >2013-03-11 17:25:00,124 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [7911aa45] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:25:00,125 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [7911aa45] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 17:25:00,127 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [7911aa45] Autorecovering 1 storage domains >2013-03-11 17:25:00,127 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [7911aa45] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 17:25:00,128 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [b7f61f0] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:25:00,128 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [b7f61f0] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:25 PM >2013-03-11 17:25:00,134 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [b7f61f0] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:25 PM >2013-03-11 17:25:02,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [60f445d8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1486243c >2013-03-11 17:25:02,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [60f445d8] FINISH, GetHardwareInfoVDSCommand, log id: 1486243c >2013-03-11 17:25:02,476 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-52) [4e72a49b] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 17:25:02,480 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-52) [3c304b3b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 17:25:02,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 5733bd82 >2013-03-11 17:25:06,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 17:25:06,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:25:06,462 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 17:25:06,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 5733bd82 >2013-03-11 17:25:06,467 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [224cc7b7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:25:06,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [224cc7b7] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 40c37607 >2013-03-11 17:25:06,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [224cc7b7] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 17:25:06,483 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [224cc7b7] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 40c37607 >2013-03-11 17:25:06,485 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [224cc7b7] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 17:25:06,485 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [3c304b3b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 17:25:06,498 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-52) [6f7a61bb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:25:06,500 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [6f7a61bb] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1f03c7e8 >2013-03-11 17:25:06,501 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [6f7a61bb] FINISH, SetVdsStatusVDSCommand, log id: 1f03c7e8 >2013-03-11 17:25:06,507 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-52) [6f7a61bb] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 17:25:06,533 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [6142e1b2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:25:06,535 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [5d18c79f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:25:06,537 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-52) [5d18c79f] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 17:25:19,216 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-11 17:25:19,224 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:25:19,225 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:26:19,218 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-11 17:26:19,224 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:26:19,224 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:27:19,220 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 17:27:19,226 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:27:19,227 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:28:19,229 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 17:28:19,235 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:28:19,235 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:29:19,224 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-11 17:29:19,231 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:29:19,231 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [1009adde] Autorecovering 1 hosts >2013-03-11 17:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [1009adde] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [5b5f4801] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [5b5f4801] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [5b5f4801] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7a2b8e7b >2013-03-11 17:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [5b5f4801] FINISH, SetVdsStatusVDSCommand, log id: 7a2b8e7b >2013-03-11 17:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [5b5f4801] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 47e69c36 >2013-03-11 17:30:00,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [5b5f4801] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1ede99ef >2013-03-11 17:30:00,086 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [5b5f4801] FINISH, GetHardwareInfoVDSCommand, log id: 1ede99ef >2013-03-11 17:30:00,103 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [185513b0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:30:00,105 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [67a6d0e1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:30:00,107 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [67a6d0e1] FINISH, ActivateVdsVDSCommand, log id: 47e69c36 >2013-03-11 17:30:00,127 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [67a6d0e1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:30:00,128 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [67a6d0e1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 17:30:00,129 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [67a6d0e1] Autorecovering 1 storage domains >2013-03-11 17:30:00,129 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [67a6d0e1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 17:30:00,130 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [626df19b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:30:00,130 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [626df19b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:30 PM >2013-03-11 17:30:00,133 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [626df19b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:30 PM >2013-03-11 17:30:00,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [483b71a0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 480f1637 >2013-03-11 17:30:00,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [483b71a0] FINISH, GetHardwareInfoVDSCommand, log id: 480f1637 >2013-03-11 17:30:00,827 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-94) [2891adab] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 17:30:00,830 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-94) [12b37c69] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 17:30:00,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 38ddb559 >2013-03-11 17:30:06,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 17:30:06,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:30:06,868 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 17:30:06,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 38ddb559 >2013-03-11 17:30:06,872 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [3704b57b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:30:06,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [3704b57b] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1383d6b9 >2013-03-11 17:30:06,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [3704b57b] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 17:30:06,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [3704b57b] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1383d6b9 >2013-03-11 17:30:06,890 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [3704b57b] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 17:30:06,891 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [12b37c69] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 17:30:06,900 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-94) [242d9847] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:30:06,902 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [242d9847] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4bc95e9c >2013-03-11 17:30:06,903 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-94) [242d9847] FINISH, SetVdsStatusVDSCommand, log id: 4bc95e9c >2013-03-11 17:30:06,910 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-94) [242d9847] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 17:30:06,938 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [7bf91a42] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:30:06,940 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [753f9c8a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:30:06,942 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-94) [753f9c8a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 17:30:19,252 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-11 17:30:19,259 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:30:19,259 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:30:33,897 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-44) [131033c6] Start refreshing all users data >2013-03-11 17:30:35,117 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-95) [7777b5f1] Attempting to update VMs/Templates Ovf. >2013-03-11 17:30:35,118 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-95) [7777b5f1] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 17:30:35,125 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-95) [7777b5f1] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 17:30:35,125 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-95) [7777b5f1] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 17:30:35,128 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-95) [7777b5f1] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 17:30:35,128 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-95) [7777b5f1] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 17:30:35,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-95) [7777b5f1] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 17:31:19,227 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 17:31:19,232 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:31:19,232 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:32:19,227 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-11 17:32:19,233 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:32:19,233 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:33:19,229 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-11 17:33:19,236 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:33:19,236 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:34:19,231 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-11 17:34:19,237 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:34:19,238 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [7d975ccd] Autorecovering 1 hosts >2013-03-11 17:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [7d975ccd] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [2a637889] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [2a637889] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-29) [2a637889] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 7fa35e91 >2013-03-11 17:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-29) [2a637889] FINISH, SetVdsStatusVDSCommand, log id: 7fa35e91 >2013-03-11 17:35:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-29) [2a637889] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 7059877b >2013-03-11 17:35:00,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-29) [2a637889] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2eb94478 >2013-03-11 17:35:00,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-29) [2a637889] FINISH, GetHardwareInfoVDSCommand, log id: 2eb94478 >2013-03-11 17:35:00,208 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-29) [7ef6148e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:35:00,210 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-29) [77a09115] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:35:00,212 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-29) [77a09115] FINISH, ActivateVdsVDSCommand, log id: 7059877b >2013-03-11 17:35:00,231 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-29) [77a09115] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:35:00,232 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-29) [77a09115] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 17:35:00,234 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [77a09115] Autorecovering 1 storage domains >2013-03-11 17:35:00,234 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-29) [77a09115] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 17:35:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [5ea98a1a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:35:00,235 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [5ea98a1a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:35 PM >2013-03-11 17:35:00,237 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-29) [5ea98a1a] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:35 PM >2013-03-11 17:35:01,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [31822876] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2695c6ec >2013-03-11 17:35:01,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-55) [31822876] FINISH, GetHardwareInfoVDSCommand, log id: 2695c6ec >2013-03-11 17:35:01,244 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-55) [5a4528df] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 17:35:01,248 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-55) [324a52f3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 17:35:01,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 242e56e8 >2013-03-11 17:35:09,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 17:35:09,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:35:09,252 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 17:35:09,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 242e56e8 >2013-03-11 17:35:09,258 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [658addfc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:35:09,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [658addfc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 63b75eea >2013-03-11 17:35:09,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [658addfc] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 17:35:09,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [658addfc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 63b75eea >2013-03-11 17:35:09,285 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [658addfc] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 17:35:09,285 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [324a52f3] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 17:35:09,296 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-55) [1737353] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:35:09,297 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [1737353] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 708ee47b >2013-03-11 17:35:09,298 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-55) [1737353] FINISH, SetVdsStatusVDSCommand, log id: 708ee47b >2013-03-11 17:35:09,304 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-55) [1737353] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 17:35:09,329 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-55) [29647d91] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:35:09,332 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-55) [27e2f0c2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:35:09,333 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-55) [27e2f0c2] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 17:35:19,233 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-11 17:35:19,240 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:35:19,240 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:36:19,234 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-11 17:36:19,239 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:36:19,240 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:37:19,241 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-11 17:37:19,254 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:37:19,254 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:38:19,238 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-11 17:38:19,245 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:38:19,245 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:39:19,239 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-11 17:39:19,245 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:39:19,245 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [ecf26e7] Autorecovering 1 hosts >2013-03-11 17:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [ecf26e7] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-5) [25ff0000] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-5) [25ff0000] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [25ff0000] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 73fdb0d2 >2013-03-11 17:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-5) [25ff0000] FINISH, SetVdsStatusVDSCommand, log id: 73fdb0d2 >2013-03-11 17:40:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-5) [25ff0000] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 77f3e930 >2013-03-11 17:40:00,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [25ff0000] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 628709a >2013-03-11 17:40:00,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-5) [25ff0000] FINISH, GetHardwareInfoVDSCommand, log id: 628709a >2013-03-11 17:40:00,261 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-5) [60de6783] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:40:00,263 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-5) [622928b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:40:00,265 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-5) [622928b] FINISH, ActivateVdsVDSCommand, log id: 77f3e930 >2013-03-11 17:40:00,284 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-5) [622928b] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:40:00,285 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-5) [622928b] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 17:40:00,287 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [622928b] Autorecovering 1 storage domains >2013-03-11 17:40:00,287 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-5) [622928b] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 17:40:00,287 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-5) [58804174] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:40:00,288 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-5) [58804174] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:40 PM >2013-03-11 17:40:00,291 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-5) [58804174] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:40 PM >2013-03-11 17:40:00,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [27385058] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 23f8a837 >2013-03-11 17:40:00,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-67) [27385058] FINISH, GetHardwareInfoVDSCommand, log id: 23f8a837 >2013-03-11 17:40:00,632 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-67) [597e5907] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 17:40:00,636 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-67) [2d9e59fe] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 17:40:00,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 6a2eed91 >2013-03-11 17:40:05,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 17:40:05,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:40:05,640 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 17:40:05,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 6a2eed91 >2013-03-11 17:40:05,644 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [eb84b5d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:40:05,658 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [eb84b5d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 57e8cae5 >2013-03-11 17:40:05,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [eb84b5d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 17:40:05,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [eb84b5d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 57e8cae5 >2013-03-11 17:40:05,661 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [eb84b5d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 17:40:05,661 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [2d9e59fe] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 17:40:05,674 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-67) [5e9f4b74] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:40:05,676 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [5e9f4b74] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7a458236 >2013-03-11 17:40:05,677 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-67) [5e9f4b74] FINISH, SetVdsStatusVDSCommand, log id: 7a458236 >2013-03-11 17:40:05,689 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-67) [5e9f4b74] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 17:40:05,718 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-67) [59e01f9d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:40:05,721 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-67) [2b47e7da] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:40:05,723 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-67) [2b47e7da] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 17:40:19,240 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-11 17:40:19,247 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:40:19,248 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:41:19,243 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-11 17:41:19,250 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:41:19,250 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:42:19,244 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 17:42:19,256 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:42:19,256 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:43:19,247 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-11 17:43:19,254 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:43:19,254 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:44:19,249 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-11 17:44:19,255 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:44:19,256 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-81) [1dc1a745] Autorecovering 1 hosts >2013-03-11 17:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-81) [1dc1a745] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-81) [131cee54] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:45:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-81) [131cee54] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:45:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [131cee54] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 791fc847 >2013-03-11 17:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-81) [131cee54] FINISH, SetVdsStatusVDSCommand, log id: 791fc847 >2013-03-11 17:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-81) [131cee54] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 21dba0bc >2013-03-11 17:45:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [131cee54] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4935f133 >2013-03-11 17:45:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-81) [131cee54] FINISH, GetHardwareInfoVDSCommand, log id: 4935f133 >2013-03-11 17:45:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-81) [654dce29] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:45:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-81) [575d20a4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:45:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-81) [575d20a4] FINISH, ActivateVdsVDSCommand, log id: 21dba0bc >2013-03-11 17:45:00,224 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-81) [575d20a4] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:45:00,225 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-81) [575d20a4] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 17:45:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-81) [575d20a4] Autorecovering 1 storage domains >2013-03-11 17:45:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-81) [575d20a4] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 17:45:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-81) [6500c96f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:45:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-81) [6500c96f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:45 PM >2013-03-11 17:45:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-81) [6500c96f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:45 PM >2013-03-11 17:45:02,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [54dc2862] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3aaa0 >2013-03-11 17:45:02,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-7) [54dc2862] FINISH, GetHardwareInfoVDSCommand, log id: 3aaa0 >2013-03-11 17:45:03,015 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-7) [2420cf61] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 17:45:03,019 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-7) [65cec06b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 17:45:03,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 51686711 >2013-03-11 17:45:10,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 17:45:10,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:45:10,006 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 17:45:10,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 51686711 >2013-03-11 17:45:10,010 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [641b5850] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:45:10,025 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [641b5850] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 29d5e6d8 >2013-03-11 17:45:10,025 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [641b5850] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 17:45:10,025 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [641b5850] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 29d5e6d8 >2013-03-11 17:45:10,028 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [641b5850] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 17:45:10,028 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [65cec06b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 17:45:10,038 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-7) [80c7762] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:45:10,040 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [80c7762] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1f5aeb03 >2013-03-11 17:45:10,041 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-7) [80c7762] FINISH, SetVdsStatusVDSCommand, log id: 1f5aeb03 >2013-03-11 17:45:10,048 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-7) [80c7762] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 17:45:10,073 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-7) [4749f45f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:45:10,076 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-7) [5bd9307d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:45:10,077 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-7) [5bd9307d] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 17:45:19,250 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-11 17:45:19,257 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:45:19,257 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:46:19,252 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-11 17:46:19,259 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:46:19,259 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:47:19,255 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-11 17:47:19,262 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:47:19,262 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:48:19,256 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-11 17:48:19,263 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:48:19,263 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:49:19,259 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-11 17:49:19,266 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:49:19,266 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [7e5f59d] Autorecovering 1 hosts >2013-03-11 17:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [7e5f59d] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [32d84f7a] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [32d84f7a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [32d84f7a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 22fe2ca9 >2013-03-11 17:50:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-92) [32d84f7a] FINISH, SetVdsStatusVDSCommand, log id: 22fe2ca9 >2013-03-11 17:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-92) [32d84f7a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 60f4d76f >2013-03-11 17:50:00,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [32d84f7a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 48a03a32 >2013-03-11 17:50:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-92) [32d84f7a] FINISH, GetHardwareInfoVDSCommand, log id: 48a03a32 >2013-03-11 17:50:00,198 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-92) [1e8cd7e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:50:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-92) [149ffb2e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:50:00,202 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-92) [149ffb2e] FINISH, ActivateVdsVDSCommand, log id: 60f4d76f >2013-03-11 17:50:00,222 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-92) [149ffb2e] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:50:00,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-92) [149ffb2e] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 17:50:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [149ffb2e] Autorecovering 1 storage domains >2013-03-11 17:50:00,224 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-92) [149ffb2e] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 17:50:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [641695f] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:50:00,225 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [641695f] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:50 PM >2013-03-11 17:50:00,228 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-92) [641695f] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:50 PM >2013-03-11 17:50:01,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [1c72fa7a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 283c84f6 >2013-03-11 17:50:01,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [1c72fa7a] FINISH, GetHardwareInfoVDSCommand, log id: 283c84f6 >2013-03-11 17:50:01,366 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-84) [58bb7dea] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 17:50:01,370 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-84) [2877902b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 17:50:01,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 317dd3e5 >2013-03-11 17:50:08,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 17:50:08,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:50:08,364 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 17:50:08,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 317dd3e5 >2013-03-11 17:50:08,367 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [3ef89c0f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:50:08,381 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [3ef89c0f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 4f5dcc6b >2013-03-11 17:50:08,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [3ef89c0f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 17:50:08,382 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [3ef89c0f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 4f5dcc6b >2013-03-11 17:50:08,384 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [3ef89c0f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 17:50:08,384 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [2877902b] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 17:50:08,397 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-84) [6a8062d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:50:08,399 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [6a8062d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6a86002b >2013-03-11 17:50:08,400 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [6a8062d] FINISH, SetVdsStatusVDSCommand, log id: 6a86002b >2013-03-11 17:50:08,407 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-84) [6a8062d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 17:50:08,434 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-84) [5a91f708] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:50:08,437 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-84) [13efd769] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:50:08,439 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-84) [13efd769] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 17:50:19,260 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-11 17:50:19,268 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:50:19,268 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:51:19,262 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 17:51:19,272 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:51:19,273 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:52:19,264 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-11 17:52:19,269 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:52:19,270 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:53:19,266 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-11 17:53:19,273 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:53:19,273 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:54:19,269 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2013-03-11 17:54:19,273 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:54:19,273 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-8) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [1fd27d46] Autorecovering 1 hosts >2013-03-11 17:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [1fd27d46] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [46aad600] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [46aad600] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [46aad600] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 23ff6302 >2013-03-11 17:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-65) [46aad600] FINISH, SetVdsStatusVDSCommand, log id: 23ff6302 >2013-03-11 17:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-65) [46aad600] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3bc547f1 >2013-03-11 17:55:00,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [46aad600] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4a68cc0b >2013-03-11 17:55:00,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-65) [46aad600] FINISH, GetHardwareInfoVDSCommand, log id: 4a68cc0b >2013-03-11 17:55:00,404 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-65) [4ac078b1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:55:00,406 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-65) [35a517cd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:55:00,408 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-65) [35a517cd] FINISH, ActivateVdsVDSCommand, log id: 3bc547f1 >2013-03-11 17:55:00,428 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-65) [35a517cd] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 17:55:00,429 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-65) [35a517cd] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 17:55:00,432 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [35a517cd] Autorecovering 1 storage domains >2013-03-11 17:55:00,432 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-65) [35a517cd] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 17:55:00,433 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [7aaf14ab] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:55:00,433 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [7aaf14ab] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 5:55 PM >2013-03-11 17:55:00,436 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-65) [7aaf14ab] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 5:55 PM >2013-03-11 17:55:02,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [14ed8f06] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: b917c74 >2013-03-11 17:55:02,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-42) [14ed8f06] FINISH, GetHardwareInfoVDSCommand, log id: b917c74 >2013-03-11 17:55:02,729 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-42) [7bef7922] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 17:55:02,732 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-42) [5591fa8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 17:55:02,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 52a78e53 >2013-03-11 17:55:09,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 17:55:09,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-47) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 17:55:09,733 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-47) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 17:55:09,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 52a78e53 >2013-03-11 17:55:09,737 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-47) [462c58db] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 17:55:09,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [462c58db] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 1c558b1d >2013-03-11 17:55:09,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-47) [462c58db] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 17:55:09,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-47) [462c58db] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 1c558b1d >2013-03-11 17:55:09,754 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-47) [462c58db] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 17:55:09,755 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [5591fa8] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 17:55:09,765 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-42) [48d80419] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:55:09,767 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [48d80419] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 69a4a07e >2013-03-11 17:55:09,768 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-42) [48d80419] FINISH, SetVdsStatusVDSCommand, log id: 69a4a07e >2013-03-11 17:55:09,774 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-42) [48d80419] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 17:55:09,801 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-42) [3e737020] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:55:09,804 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-42) [69cf856a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 17:55:09,805 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-42) [69cf856a] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 17:55:19,270 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-11 17:55:19,277 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:55:19,277 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:56:19,271 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-11 17:56:19,279 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:56:19,279 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:57:19,273 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-11 17:57:19,289 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:57:19,289 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:58:19,275 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 17:58:19,282 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:58:19,282 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 17:59:19,277 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-11 17:59:19,284 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 17:59:19,284 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) [617028df] Autorecovering 1 hosts >2013-03-11 18:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) [617028df] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-61) [2d09ebc1] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-61) [2d09ebc1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [2d09ebc1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1d8f3d59 >2013-03-11 18:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-61) [2d09ebc1] FINISH, SetVdsStatusVDSCommand, log id: 1d8f3d59 >2013-03-11 18:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-61) [2d09ebc1] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 73d580f7 >2013-03-11 18:00:00,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [2d09ebc1] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3ce70429 >2013-03-11 18:00:00,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-61) [2d09ebc1] FINISH, GetHardwareInfoVDSCommand, log id: 3ce70429 >2013-03-11 18:00:00,221 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-61) [66aedbdb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:00:00,223 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-61) [217e2e36] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:00:00,225 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-61) [217e2e36] FINISH, ActivateVdsVDSCommand, log id: 73d580f7 >2013-03-11 18:00:00,245 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-61) [217e2e36] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:00:00,246 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-61) [217e2e36] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 18:00:00,247 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) [217e2e36] Autorecovering 1 storage domains >2013-03-11 18:00:00,247 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-61) [217e2e36] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 18:00:00,248 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-61) [57114d6c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:00:00,248 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-61) [57114d6c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:00 PM >2013-03-11 18:00:00,251 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-61) [57114d6c] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:00 PM >2013-03-11 18:00:01,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [2e4e7034] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4dca1f2e >2013-03-11 18:00:01,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-72) [2e4e7034] FINISH, GetHardwareInfoVDSCommand, log id: 4dca1f2e >2013-03-11 18:00:01,095 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-72) [39f564be] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 18:00:01,099 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-72) [3d7a0a3a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 18:00:01,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 60c1e425 >2013-03-11 18:00:07,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 18:00:07,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:00:07,135 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 18:00:07,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 60c1e425 >2013-03-11 18:00:07,139 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [51f7117d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:00:07,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [51f7117d] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 61192fce >2013-03-11 18:00:07,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [51f7117d] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 18:00:07,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [51f7117d] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 61192fce >2013-03-11 18:00:07,156 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [51f7117d] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 18:00:07,156 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [3d7a0a3a] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 18:00:07,166 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-72) [3ca65f6d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:00:07,167 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [3ca65f6d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 92aa17b >2013-03-11 18:00:07,168 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-72) [3ca65f6d] FINISH, SetVdsStatusVDSCommand, log id: 92aa17b >2013-03-11 18:00:07,174 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-72) [3ca65f6d] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 18:00:07,199 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-72) [13765dc9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:00:07,201 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-72) [133e5ad0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:00:07,203 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-72) [133e5ad0] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 18:00:19,278 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-11 18:00:19,285 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:00:19,285 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:01:19,280 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-11 18:01:19,286 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:01:19,287 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:02:19,282 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-11 18:02:19,289 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:02:19,290 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:02:54,617 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (WebAdminUpperRightButtonURL) with null version, using default general for version >2013-03-11 18:03:52,607 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 18:03:52,614 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:03:52,614 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:04:52,608 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-11 18:04:52,614 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:04:52,614 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:05:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [32acbdc8] Autorecovering 1 hosts >2013-03-11 18:05:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [32acbdc8] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [96c3ba7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [96c3ba7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [96c3ba7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6eaeb405 >2013-03-11 18:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-19) [96c3ba7] FINISH, SetVdsStatusVDSCommand, log id: 6eaeb405 >2013-03-11 18:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-19) [96c3ba7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6cc2e636 >2013-03-11 18:05:00,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [96c3ba7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 338db570 >2013-03-11 18:05:00,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-19) [96c3ba7] FINISH, GetHardwareInfoVDSCommand, log id: 338db570 >2013-03-11 18:05:00,200 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-19) [35095f7e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:05:00,203 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-19) [471b7dd8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:05:00,204 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-19) [471b7dd8] FINISH, ActivateVdsVDSCommand, log id: 6cc2e636 >2013-03-11 18:05:00,224 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-19) [471b7dd8] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:05:00,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-19) [471b7dd8] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 18:05:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [471b7dd8] Autorecovering 1 storage domains >2013-03-11 18:05:00,226 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-19) [471b7dd8] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 18:05:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [16014079] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:05:00,227 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [16014079] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:05 PM >2013-03-11 18:05:00,230 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-19) [16014079] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:05 PM >2013-03-11 18:05:01,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [520adeb3] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2b0a79af >2013-03-11 18:05:01,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [520adeb3] FINISH, GetHardwareInfoVDSCommand, log id: 2b0a79af >2013-03-11 18:05:01,484 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-49) [5b2e708a] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 18:05:01,487 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-49) [a309978] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 18:05:01,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2ed092c4 >2013-03-11 18:05:07,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 18:05:07,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:05:07,525 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 18:05:07,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2ed092c4 >2013-03-11 18:05:07,529 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [68eb12dc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:05:07,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [68eb12dc] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 6734ba0e >2013-03-11 18:05:07,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [68eb12dc] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 18:05:07,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [68eb12dc] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 6734ba0e >2013-03-11 18:05:07,550 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [68eb12dc] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 18:05:07,550 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [a309978] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 18:05:07,561 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-49) [78640b39] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:05:07,562 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [78640b39] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 66e52e7d >2013-03-11 18:05:07,563 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [78640b39] FINISH, SetVdsStatusVDSCommand, log id: 66e52e7d >2013-03-11 18:05:07,568 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-49) [78640b39] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 18:05:07,594 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-49) [38fb7b11] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:05:07,598 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-49) [72fbc680] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:05:07,600 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) [72fbc680] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 18:05:52,610 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-11 18:05:52,618 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:05:52,618 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:06:52,610 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-11 18:06:52,622 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:06:52,622 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:07:52,633 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-11 18:07:52,640 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:07:52,640 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:08:52,620 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-11 18:08:52,625 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:08:52,626 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:09:52,617 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-11 18:09:52,624 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:09:52,624 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [7845ae28] Autorecovering 1 hosts >2013-03-11 18:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [7845ae28] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-28) [12cf96a8] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:10:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-28) [12cf96a8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [12cf96a8] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4b52656b >2013-03-11 18:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-28) [12cf96a8] FINISH, SetVdsStatusVDSCommand, log id: 4b52656b >2013-03-11 18:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-28) [12cf96a8] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 77390140 >2013-03-11 18:10:00,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [12cf96a8] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 186c9e56 >2013-03-11 18:10:00,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [12cf96a8] FINISH, GetHardwareInfoVDSCommand, log id: 186c9e56 >2013-03-11 18:10:00,220 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-28) [f7ada9a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:10:00,223 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-28) [138ed694] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:10:00,224 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-28) [138ed694] FINISH, ActivateVdsVDSCommand, log id: 77390140 >2013-03-11 18:10:00,244 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-28) [138ed694] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:10:00,245 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-28) [138ed694] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 18:10:00,246 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [138ed694] Autorecovering 1 storage domains >2013-03-11 18:10:00,246 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-28) [138ed694] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 18:10:00,247 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-28) [9b9714b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:10:00,247 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-28) [9b9714b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:10 PM >2013-03-11 18:10:00,250 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-28) [9b9714b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:10 PM >2013-03-11 18:10:01,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [12ed4689] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 7deaf175 >2013-03-11 18:10:01,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-79) [12ed4689] FINISH, GetHardwareInfoVDSCommand, log id: 7deaf175 >2013-03-11 18:10:01,903 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-79) [7b76aa19] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 18:10:01,906 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-79) [48d50f66] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 18:10:01,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 2d19d34d >2013-03-11 18:10:08,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 18:10:08,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:10:08,903 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 18:10:08,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2d19d34d >2013-03-11 18:10:08,906 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [76c53d2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:10:08,920 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [76c53d2] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 65be74cd >2013-03-11 18:10:08,920 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [76c53d2] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 18:10:08,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [76c53d2] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 65be74cd >2013-03-11 18:10:08,923 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [76c53d2] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 18:10:08,923 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [48d50f66] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 18:10:08,933 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-79) [7f83ff6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:10:08,935 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [7f83ff6] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 17451508 >2013-03-11 18:10:08,936 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-79) [7f83ff6] FINISH, SetVdsStatusVDSCommand, log id: 17451508 >2013-03-11 18:10:08,942 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-79) [7f83ff6] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 18:10:08,968 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-79) [b7daee1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:10:08,970 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-79) [23b10d31] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:10:08,972 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-79) [23b10d31] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 18:10:52,618 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 18:10:52,632 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:10:52,633 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:11:52,621 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-11 18:11:52,628 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:11:52,628 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:12:52,622 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 18:12:52,629 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:12:52,629 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:13:52,625 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-11 18:13:52,631 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:13:52,631 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:14:52,625 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 18:14:52,631 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:14:52,632 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:15:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [58279a55] Autorecovering 1 hosts >2013-03-11 18:15:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [58279a55] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [55116642] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [55116642] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [55116642] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 6169c129 >2013-03-11 18:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-13) [55116642] FINISH, SetVdsStatusVDSCommand, log id: 6169c129 >2013-03-11 18:15:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-13) [55116642] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 5942e9eb >2013-03-11 18:15:00,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [55116642] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 574a5bed >2013-03-11 18:15:00,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-13) [55116642] FINISH, GetHardwareInfoVDSCommand, log id: 574a5bed >2013-03-11 18:15:00,204 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-13) [285955dc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:15:00,207 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-13) [4cc48775] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:15:00,208 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-13) [4cc48775] FINISH, ActivateVdsVDSCommand, log id: 5942e9eb >2013-03-11 18:15:00,228 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-13) [4cc48775] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:15:00,229 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-13) [4cc48775] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 18:15:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [4cc48775] Autorecovering 1 storage domains >2013-03-11 18:15:00,231 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-13) [4cc48775] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 18:15:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [3df04d79] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:15:00,232 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [3df04d79] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:15 PM >2013-03-11 18:15:00,238 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-13) [3df04d79] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:15 PM >2013-03-11 18:15:00,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [725358bc] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4172c4da >2013-03-11 18:15:00,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-34) [725358bc] FINISH, GetHardwareInfoVDSCommand, log id: 4172c4da >2013-03-11 18:15:00,308 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-34) [1b2a04ac] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 18:15:00,311 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-34) [3c24f814] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 18:15:00,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 68c85d01 >2013-03-11 18:15:05,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 18:15:05,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:15:05,304 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 18:15:05,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 68c85d01 >2013-03-11 18:15:05,308 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [9bc3c06] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:15:05,322 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [9bc3c06] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 510a4998 >2013-03-11 18:15:05,322 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [9bc3c06] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 18:15:05,322 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [9bc3c06] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 510a4998 >2013-03-11 18:15:05,324 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [9bc3c06] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 18:15:05,324 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [3c24f814] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 18:15:05,335 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-34) [114f6127] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:15:05,336 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [114f6127] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 26c8c2c5 >2013-03-11 18:15:05,338 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-34) [114f6127] FINISH, SetVdsStatusVDSCommand, log id: 26c8c2c5 >2013-03-11 18:15:05,344 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-34) [114f6127] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 18:15:05,410 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-34) [56f72cc6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:15:05,412 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-34) [13cac3c6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:15:05,414 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) [13cac3c6] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 18:15:52,626 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-11 18:15:52,634 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:15:52,634 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:16:52,628 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-11 18:16:52,634 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:16:52,635 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:17:52,658 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-11 18:17:52,665 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:17:52,665 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:18:52,632 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-11 18:18:52,639 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:18:52,639 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:19:52,633 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-11 18:19:52,640 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:19:52,640 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [755df19b] Autorecovering 1 hosts >2013-03-11 18:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [755df19b] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [26ee9649] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [26ee9649] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [26ee9649] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 42aa2054 >2013-03-11 18:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [26ee9649] FINISH, SetVdsStatusVDSCommand, log id: 42aa2054 >2013-03-11 18:20:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-46) [26ee9649] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: cb4f17a >2013-03-11 18:20:00,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [26ee9649] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2fd2e361 >2013-03-11 18:20:00,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [26ee9649] FINISH, GetHardwareInfoVDSCommand, log id: 2fd2e361 >2013-03-11 18:20:00,456 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-46) [3ce93cb9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:20:00,459 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-46) [2c2453f1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:20:00,460 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-46) [2c2453f1] FINISH, ActivateVdsVDSCommand, log id: cb4f17a >2013-03-11 18:20:00,479 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-46) [2c2453f1] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:20:00,480 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-46) [2c2453f1] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 18:20:00,482 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [2c2453f1] Autorecovering 1 storage domains >2013-03-11 18:20:00,482 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-46) [2c2453f1] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 18:20:00,483 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [13507658] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:20:00,483 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [13507658] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:20 PM >2013-03-11 18:20:00,489 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-46) [13507658] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:20 PM >2013-03-11 18:20:02,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [53af3509] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1a62317 >2013-03-11 18:20:02,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-75) [53af3509] FINISH, GetHardwareInfoVDSCommand, log id: 1a62317 >2013-03-11 18:20:02,715 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-75) [e901900] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 18:20:02,719 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-75) [70704b1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 18:20:02,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 71e004c8 >2013-03-11 18:20:08,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 18:20:08,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:20:08,710 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 18:20:08,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 71e004c8 >2013-03-11 18:20:08,714 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [464eb154] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:20:08,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [464eb154] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 7c747876 >2013-03-11 18:20:08,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [464eb154] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 18:20:08,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [464eb154] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 7c747876 >2013-03-11 18:20:08,733 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-47) [70704b1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 18:20:08,744 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-75) [1656ccc1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:20:08,745 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [1656ccc1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 37ef851a >2013-03-11 18:20:08,747 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-75) [1656ccc1] FINISH, SetVdsStatusVDSCommand, log id: 37ef851a >2013-03-11 18:20:08,752 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-75) [1656ccc1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 18:20:08,783 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-75) [47a5eef9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:20:08,786 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-75) [33b57fd7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:20:08,787 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-75) [33b57fd7] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 18:20:52,636 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-11 18:20:52,644 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:20:52,645 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:21:52,638 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-11 18:21:52,645 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:21:52,645 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:22:52,641 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-11 18:22:52,650 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:22:52,650 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:23:52,641 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-11 18:23:52,648 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:23:52,648 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:24:52,644 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-11 18:24:52,651 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:24:52,651 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:25:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [5ea767a] Autorecovering 1 hosts >2013-03-11 18:25:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [5ea767a] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [487f6fb0] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [487f6fb0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [487f6fb0] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: da62f98 >2013-03-11 18:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-22) [487f6fb0] FINISH, SetVdsStatusVDSCommand, log id: da62f98 >2013-03-11 18:25:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-22) [487f6fb0] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6943ba1e >2013-03-11 18:25:00,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [487f6fb0] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 33d5cab >2013-03-11 18:25:00,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-22) [487f6fb0] FINISH, GetHardwareInfoVDSCommand, log id: 33d5cab >2013-03-11 18:25:00,091 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-22) [22ed5b1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:25:00,094 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-22) [1dcb4ee5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:25:00,095 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-22) [1dcb4ee5] FINISH, ActivateVdsVDSCommand, log id: 6943ba1e >2013-03-11 18:25:00,115 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-22) [1dcb4ee5] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:25:00,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-22) [1dcb4ee5] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 18:25:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [1dcb4ee5] Autorecovering 1 storage domains >2013-03-11 18:25:00,118 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-22) [1dcb4ee5] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 18:25:00,118 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [39deeb81] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:25:00,119 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [39deeb81] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:25 PM >2013-03-11 18:25:00,124 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-22) [39deeb81] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:25 PM >2013-03-11 18:25:03,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [6ba4602a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 62d0dd2d >2013-03-11 18:25:03,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-56) [6ba4602a] FINISH, GetHardwareInfoVDSCommand, log id: 62d0dd2d >2013-03-11 18:25:03,088 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-56) [7df26840] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 18:25:03,092 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-56) [3ed16477] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 18:25:03,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4942c4d8 >2013-03-11 18:25:09,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 18:25:09,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:25:09,077 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 18:25:09,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 4942c4d8 >2013-03-11 18:25:09,081 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [3fb0516f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:25:09,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [3fb0516f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 35011230 >2013-03-11 18:25:09,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [3fb0516f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 18:25:09,095 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [3fb0516f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 35011230 >2013-03-11 18:25:09,097 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [3fb0516f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 18:25:09,098 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [3ed16477] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 18:25:09,108 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-56) [2ab7e570] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:25:09,109 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [2ab7e570] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: ea8649f >2013-03-11 18:25:09,111 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-56) [2ab7e570] FINISH, SetVdsStatusVDSCommand, log id: ea8649f >2013-03-11 18:25:09,117 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-56) [2ab7e570] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 18:25:09,143 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-56) [74782b07] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:25:09,145 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-56) [25df1138] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:25:09,147 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-56) [25df1138] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 18:25:52,645 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 18:25:52,652 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:25:52,652 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:26:52,646 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-11 18:26:52,653 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:26:52,653 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:27:52,647 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-11 18:27:52,654 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:27:52,654 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:28:52,650 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-11 18:28:52,657 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:28:52,657 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:29:52,651 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-11 18:29:52,661 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:29:52,661 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [376e74e2] Autorecovering 1 hosts >2013-03-11 18:30:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [376e74e2] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [34c7d842] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:30:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [34c7d842] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [34c7d842] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 1520713f >2013-03-11 18:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [34c7d842] FINISH, SetVdsStatusVDSCommand, log id: 1520713f >2013-03-11 18:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [34c7d842] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: bf7cbd5 >2013-03-11 18:30:00,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [34c7d842] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: ef171d8 >2013-03-11 18:30:00,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [34c7d842] FINISH, GetHardwareInfoVDSCommand, log id: ef171d8 >2013-03-11 18:30:00,097 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [3bc2880a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:30:00,100 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [bd78358] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:30:00,101 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-8) [bd78358] FINISH, ActivateVdsVDSCommand, log id: bf7cbd5 >2013-03-11 18:30:00,120 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-8) [bd78358] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:30:00,121 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-8) [bd78358] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 18:30:00,123 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [bd78358] Autorecovering 1 storage domains >2013-03-11 18:30:00,123 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-8) [bd78358] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 18:30:00,123 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [1394fb2e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:30:00,124 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [1394fb2e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:30 PM >2013-03-11 18:30:00,127 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-8) [1394fb2e] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:30 PM >2013-03-11 18:30:00,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-23) [6521f7b4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 1c460f1d >2013-03-11 18:30:00,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-23) [6521f7b4] FINISH, GetHardwareInfoVDSCommand, log id: 1c460f1d >2013-03-11 18:30:00,441 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-23) [5238e579] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 18:30:00,445 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-23) [65f17dae] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 18:30:00,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 25799827 >2013-03-11 18:30:08,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 18:30:08,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:30:08,493 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 18:30:08,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 25799827 >2013-03-11 18:30:08,497 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [199805b4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:30:08,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [199805b4] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 157201e8 >2013-03-11 18:30:08,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [199805b4] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 18:30:08,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [199805b4] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 157201e8 >2013-03-11 18:30:08,513 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [199805b4] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 18:30:08,513 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [65f17dae] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 18:30:08,524 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-23) [3b2b6e31] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:30:08,525 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-23) [3b2b6e31] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: cc0a138 >2013-03-11 18:30:08,526 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-23) [3b2b6e31] FINISH, SetVdsStatusVDSCommand, log id: cc0a138 >2013-03-11 18:30:08,533 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-23) [3b2b6e31] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 18:30:08,558 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-23) [68776423] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:30:08,561 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-23) [2ffd4573] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:30:08,562 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-23) [2ffd4573] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 18:30:33,899 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-65) [7aaf14ab] Start refreshing all users data >2013-03-11 18:30:35,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [fae2a73] Attempting to update VMs/Templates Ovf. >2013-03-11 18:30:35,131 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [fae2a73] Attempting to update VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 18:30:35,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [fae2a73] Successfully updated VM OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 18:30:35,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [fae2a73] Attempting to update template OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 18:30:35,138 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [fae2a73] Succesfully updated templates OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 18:30:35,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [fae2a73] Attempting to remove uneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 18:30:35,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-66) [fae2a73] Succesfully removed unneeded template/vm OVFs in Data Center cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 18:30:52,652 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-11 18:30:52,659 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:30:52,659 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:31:52,653 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-11 18:31:52,659 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:31:52,660 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:32:52,655 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-11 18:32:52,662 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:32:52,662 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:33:52,657 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 18:33:52,664 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:33:52,664 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:34:52,660 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-11 18:34:52,667 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:34:52,667 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [5d18c79f] Autorecovering 1 hosts >2013-03-11 18:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [5d18c79f] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [eefd279] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [eefd279] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [eefd279] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 46218593 >2013-03-11 18:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-52) [eefd279] FINISH, SetVdsStatusVDSCommand, log id: 46218593 >2013-03-11 18:35:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [eefd279] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3bdb9ca9 >2013-03-11 18:35:00,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [eefd279] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 6e29a0e3 >2013-03-11 18:35:00,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-52) [eefd279] FINISH, GetHardwareInfoVDSCommand, log id: 6e29a0e3 >2013-03-11 18:35:00,222 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-52) [2afab92f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:35:00,224 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-52) [77b99a13] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:35:00,226 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-52) [77b99a13] FINISH, ActivateVdsVDSCommand, log id: 3bdb9ca9 >2013-03-11 18:35:00,245 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-52) [77b99a13] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:35:00,246 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-52) [77b99a13] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 18:35:00,247 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [77b99a13] Autorecovering 1 storage domains >2013-03-11 18:35:00,247 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-52) [77b99a13] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 18:35:00,248 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [3b22dc02] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:35:00,248 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [3b22dc02] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:35 PM >2013-03-11 18:35:00,252 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-52) [3b22dc02] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:35 PM >2013-03-11 18:35:02,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [13507658] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3d77006 >2013-03-11 18:35:02,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-46) [13507658] FINISH, GetHardwareInfoVDSCommand, log id: 3d77006 >2013-03-11 18:35:02,861 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-46) [6ad81477] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 18:35:02,865 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-46) [35043ff1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 18:35:02,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 464570ad >2013-03-11 18:35:09,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 18:35:09,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:35:09,845 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 18:35:09,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 464570ad >2013-03-11 18:35:09,849 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [59b61aec] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:35:09,863 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [59b61aec] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 3b54a11e >2013-03-11 18:35:09,863 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [59b61aec] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 18:35:09,863 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [59b61aec] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 3b54a11e >2013-03-11 18:35:09,865 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [59b61aec] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 18:35:09,866 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [35043ff1] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 18:35:09,876 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-46) [21898cd1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:35:09,878 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [21898cd1] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 265cdc5a >2013-03-11 18:35:09,879 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-46) [21898cd1] FINISH, SetVdsStatusVDSCommand, log id: 265cdc5a >2013-03-11 18:35:09,886 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-46) [21898cd1] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 18:35:09,913 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-46) [2f6370ff] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:35:09,915 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-46) [54282698] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:35:09,916 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-46) [54282698] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 18:35:52,662 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-11 18:35:52,670 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:35:52,670 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:36:52,664 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-11 18:36:52,671 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:36:52,672 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:37:52,664 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-11 18:37:52,670 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:37:52,671 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:38:52,666 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-11 18:38:52,672 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:38:52,672 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:39:52,667 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 18:39:52,673 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:39:52,674 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [240e1971] Autorecovering 1 hosts >2013-03-11 18:40:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [240e1971] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [3fb7d099] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [3fb7d099] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [3fb7d099] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 4fd56b8e >2013-03-11 18:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-18) [3fb7d099] FINISH, SetVdsStatusVDSCommand, log id: 4fd56b8e >2013-03-11 18:40:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-18) [3fb7d099] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 16f69927 >2013-03-11 18:40:00,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [3fb7d099] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 25e0daaa >2013-03-11 18:40:00,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-18) [3fb7d099] FINISH, GetHardwareInfoVDSCommand, log id: 25e0daaa >2013-03-11 18:40:00,413 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-18) [78064665] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:40:00,416 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-18) [3051efbe] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:40:00,417 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-18) [3051efbe] FINISH, ActivateVdsVDSCommand, log id: 16f69927 >2013-03-11 18:40:00,438 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-18) [3051efbe] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:40:00,439 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-18) [3051efbe] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 18:40:00,440 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [3051efbe] Autorecovering 1 storage domains >2013-03-11 18:40:00,440 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-18) [3051efbe] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 18:40:00,441 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [7792ae75] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:40:00,441 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [7792ae75] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:40 PM >2013-03-11 18:40:00,444 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-18) [7792ae75] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:40 PM >2013-03-11 18:40:01,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [d9c8cd4] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5a4e3f3a >2013-03-11 18:40:01,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-57) [d9c8cd4] FINISH, GetHardwareInfoVDSCommand, log id: 5a4e3f3a >2013-03-11 18:40:01,215 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-57) [660155e1] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 18:40:01,218 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-57) [49b325d7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 18:40:01,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1812bf91 >2013-03-11 18:40:09,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 18:40:09,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:40:09,226 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 18:40:09,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 1812bf91 >2013-03-11 18:40:09,230 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [1d465f9f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:40:09,244 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [1d465f9f] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 11a7167f >2013-03-11 18:40:09,244 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [1d465f9f] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 18:40:09,244 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [1d465f9f] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 11a7167f >2013-03-11 18:40:09,246 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [1d465f9f] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 18:40:09,247 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [49b325d7] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 18:40:09,257 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-57) [15394939] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:40:09,259 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [15394939] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 42de638b >2013-03-11 18:40:09,260 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-57) [15394939] FINISH, SetVdsStatusVDSCommand, log id: 42de638b >2013-03-11 18:40:09,266 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-57) [15394939] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 18:40:09,291 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-57) [44354f23] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:40:09,294 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-57) [af1b9f7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:40:09,295 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-57) [af1b9f7] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 18:40:52,669 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-11 18:40:52,677 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:40:52,677 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:41:52,669 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-11 18:41:52,677 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:41:52,677 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:42:52,671 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-11 18:42:52,677 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:42:52,677 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:43:52,672 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-11 18:43:52,679 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:43:52,679 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:44:52,673 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-11 18:44:52,680 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:44:52,684 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [13efd769] Autorecovering 1 hosts >2013-03-11 18:45:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [13efd769] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [3a60a77] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [3a60a77] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [3a60a77] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 5c953cdf >2013-03-11 18:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-84) [3a60a77] FINISH, SetVdsStatusVDSCommand, log id: 5c953cdf >2013-03-11 18:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [3a60a77] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: a1557b5 >2013-03-11 18:45:00,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [3a60a77] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3e54423b >2013-03-11 18:45:00,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-84) [3a60a77] FINISH, GetHardwareInfoVDSCommand, log id: 3e54423b >2013-03-11 18:45:00,186 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-84) [61aa2b7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:45:00,189 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-84) [1f1f2c38] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:45:00,190 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-84) [1f1f2c38] FINISH, ActivateVdsVDSCommand, log id: a1557b5 >2013-03-11 18:45:00,210 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-84) [1f1f2c38] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:45:00,211 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-84) [1f1f2c38] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 18:45:00,212 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [1f1f2c38] Autorecovering 1 storage domains >2013-03-11 18:45:00,212 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-84) [1f1f2c38] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 18:45:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [3a7e32d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:45:00,213 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [3a7e32d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:45 PM >2013-03-11 18:45:00,217 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-84) [3a7e32d] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:45 PM >2013-03-11 18:45:00,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [1394fb2e] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 75f8d3bb >2013-03-11 18:45:00,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-8) [1394fb2e] FINISH, GetHardwareInfoVDSCommand, log id: 75f8d3bb >2013-03-11 18:45:00,595 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-8) [5a16a635] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 18:45:00,599 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-8) [71d25edf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 18:45:00,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 1ca25047 >2013-03-11 18:45:08,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 18:45:08,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:45:08,922 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 18:45:08,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 1ca25047 >2013-03-11 18:45:08,925 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [5916010a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:45:08,939 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5916010a] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: bb1248d >2013-03-11 18:45:08,939 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [5916010a] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 18:45:08,939 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [5916010a] FINISH, MarkPoolInReconstructModeVDSCommand, log id: bb1248d >2013-03-11 18:45:08,942 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [5916010a] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 18:45:08,942 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [71d25edf] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 18:45:08,952 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-8) [7f15e5fd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:45:08,954 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [7f15e5fd] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 21f3c584 >2013-03-11 18:45:08,955 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-8) [7f15e5fd] FINISH, SetVdsStatusVDSCommand, log id: 21f3c584 >2013-03-11 18:45:08,961 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-8) [7f15e5fd] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 18:45:08,986 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-8) [43974f8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:45:08,989 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-8) [18e591da] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:45:08,990 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-8) [18e591da] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 18:45:52,675 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-11 18:45:52,683 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:45:52,683 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:46:52,675 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-11 18:46:52,682 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:46:52,683 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:47:52,677 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-11 18:47:52,684 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:47:52,684 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:48:52,678 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 18:48:52,687 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:48:52,687 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:49:52,680 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-11 18:49:52,686 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:49:52,686 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [b7f61f0] Autorecovering 1 hosts >2013-03-11 18:50:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [b7f61f0] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [3fe300d2] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:50:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [3fe300d2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [3fe300d2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 2a3fd255 >2013-03-11 18:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-54) [3fe300d2] FINISH, SetVdsStatusVDSCommand, log id: 2a3fd255 >2013-03-11 18:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [3fe300d2] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3155bd21 >2013-03-11 18:50:00,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [3fe300d2] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4d122be2 >2013-03-11 18:50:00,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-54) [3fe300d2] FINISH, GetHardwareInfoVDSCommand, log id: 4d122be2 >2013-03-11 18:50:00,265 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-54) [4ac7d819] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:50:00,267 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-54) [6b7bcd67] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:50:00,269 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-54) [6b7bcd67] FINISH, ActivateVdsVDSCommand, log id: 3155bd21 >2013-03-11 18:50:00,288 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-54) [6b7bcd67] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:50:00,289 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-54) [6b7bcd67] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 18:50:00,290 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [6b7bcd67] Autorecovering 1 storage domains >2013-03-11 18:50:00,290 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-54) [6b7bcd67] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 18:50:00,291 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [731b8a29] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:50:00,291 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [731b8a29] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:50 PM >2013-03-11 18:50:00,294 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-54) [731b8a29] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:50 PM >2013-03-11 18:50:00,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [15116a36] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 242109b6 >2013-03-11 18:50:00,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-14) [15116a36] FINISH, GetHardwareInfoVDSCommand, log id: 242109b6 >2013-03-11 18:50:00,369 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-14) [23225fe9] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 18:50:00,373 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-14) [6226803e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 18:50:00,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 7adbc867 >2013-03-11 18:50:05,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 18:50:05,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-50) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:50:05,362 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-50) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 18:50:05,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 7adbc867 >2013-03-11 18:50:05,366 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-50) [1a78cff5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:50:05,380 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [1a78cff5] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 72eb91c4 >2013-03-11 18:50:05,380 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) [1a78cff5] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 18:50:05,380 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-50) [1a78cff5] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 72eb91c4 >2013-03-11 18:50:05,382 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-50) [1a78cff5] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 18:50:05,382 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [6226803e] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 18:50:05,392 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-14) [3e7a1691] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:50:05,394 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [3e7a1691] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 1e36e8db >2013-03-11 18:50:05,395 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-14) [3e7a1691] FINISH, SetVdsStatusVDSCommand, log id: 1e36e8db >2013-03-11 18:50:05,401 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-14) [3e7a1691] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 18:50:05,439 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-14) [2890857d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:50:05,442 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-14) [7e588f84] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:50:05,443 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-14) [7e588f84] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 18:50:52,681 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 18:50:52,687 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:50:52,688 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:51:52,684 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-11 18:51:52,691 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:51:52,692 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:52:52,685 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-11 18:52:52,691 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:52:52,692 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:53:52,686 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-11 18:53:52,693 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:53:52,693 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:54:52,686 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-11 18:54:52,694 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:54:52,694 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [626df19b] Autorecovering 1 hosts >2013-03-11 18:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [626df19b] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [4f0a5a6c] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [4f0a5a6c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [4f0a5a6c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 76ada17c >2013-03-11 18:55:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-36) [4f0a5a6c] FINISH, SetVdsStatusVDSCommand, log id: 76ada17c >2013-03-11 18:55:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [4f0a5a6c] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 3b029540 >2013-03-11 18:55:00,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [4f0a5a6c] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4c4733b >2013-03-11 18:55:00,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-36) [4f0a5a6c] FINISH, GetHardwareInfoVDSCommand, log id: 4c4733b >2013-03-11 18:55:00,225 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-36) [7d76a01b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:55:00,228 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-36) [51607276] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:55:00,230 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-36) [51607276] FINISH, ActivateVdsVDSCommand, log id: 3b029540 >2013-03-11 18:55:00,254 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-36) [51607276] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 18:55:00,255 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-36) [51607276] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 18:55:00,256 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [51607276] Autorecovering 1 storage domains >2013-03-11 18:55:00,256 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-36) [51607276] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 18:55:00,259 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [22a0bb6b] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:55:00,259 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [22a0bb6b] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 6:55 PM >2013-03-11 18:55:00,263 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-36) [22a0bb6b] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 6:55 PM >2013-03-11 18:55:02,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [72b0e656] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 5d58f13a >2013-03-11 18:55:02,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-59) [72b0e656] FINISH, GetHardwareInfoVDSCommand, log id: 5d58f13a >2013-03-11 18:55:02,752 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-59) [3e1afcdb] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 18:55:02,756 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-59) [512160ed] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 18:55:02,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4ead8bb >2013-03-11 18:55:09,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 18:55:09,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-46) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 18:55:09,775 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-46) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 18:55:09,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 4ead8bb >2013-03-11 18:55:09,779 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-46) [4163d8e1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 18:55:09,794 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [4163d8e1] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 28633621 >2013-03-11 18:55:09,794 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [4163d8e1] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 18:55:09,794 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-46) [4163d8e1] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 28633621 >2013-03-11 18:55:09,796 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-46) [4163d8e1] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 18:55:09,797 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-50) [512160ed] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 18:55:09,812 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-59) [14f6c8d2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:55:09,813 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [14f6c8d2] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 7a65537e >2013-03-11 18:55:09,815 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-59) [14f6c8d2] FINISH, SetVdsStatusVDSCommand, log id: 7a65537e >2013-03-11 18:55:09,825 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-59) [14f6c8d2] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 18:55:09,853 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-59) [296c932a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:55:09,856 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-59) [3d4e1575] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 18:55:09,857 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-59) [3d4e1575] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 18:55:52,688 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-11 18:55:52,695 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:55:52,695 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:56:52,690 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-11 18:56:52,696 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:56:52,696 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:57:52,688 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-11 18:57:52,695 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:57:52,695 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:58:19,826 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp-/127.0.0.1:8702-10) Lock Acquired to object EngineLock [exclusiveLocks= key: af0d05f3-f358-4df5-ac81-35684017b815 value: POOL >key: 77c935fb-bd6e-4f85-84e0-2377bf89e0b8 value: STORAGE >, sharedLocks= ] >2013-03-11 18:58:19,875 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-3-thread-46) [1b011f22] Running command: DeactivateStorageDomainCommand internal: false. Entities affected : ID: 77c935fb-bd6e-4f85-84e0-2377bf89e0b8 Type: Storage >2013-03-11 18:58:19,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-3-thread-46) [1b011f22] START, DeactivateStorageDomainVDSCommand( storagePoolId = af0d05f3-f358-4df5-ac81-35684017b815, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 77c935fb-bd6e-4f85-84e0-2377bf89e0b8, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 1d0f36f6 >2013-03-11 18:58:19,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-3-thread-46) [1b011f22] FINISH, DeactivateStorageDomainVDSCommand, log id: 1d0f36f6 >2013-03-11 18:58:19,921 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-3-thread-46) [1b011f22] Lock freed to object EngineLock [exclusiveLocks= key: af0d05f3-f358-4df5-ac81-35684017b815 value: POOL >key: 77c935fb-bd6e-4f85-84e0-2377bf89e0b8 value: STORAGE >, sharedLocks= ] >2013-03-11 18:58:19,924 INFO [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (pool-3-thread-45) [1b011f22] After deactivate treatment vds: cheetah01.scl.lab.tlv.redhat.com,pool cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 18:58:19,925 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (pool-3-thread-46) [1b011f22] START, SpmStopOnIrsVDSCommand( storagePoolId = af0d05f3-f358-4df5-ac81-35684017b815, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 176d0fb8 >2013-03-11 18:58:19,925 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (pool-3-thread-46) [1b011f22] START, ResetIrsVDSCommand( storagePoolId = af0d05f3-f358-4df5-ac81-35684017b815, ignoreFailoverLimit = false, compatabilityVersion = null, vdsId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, ignoreStopFailed = false), log id: 7923050b >2013-03-11 18:58:19,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-3-thread-46) [1b011f22] START, SpmStopVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = af0d05f3-f358-4df5-ac81-35684017b815), log id: 626f7463 >2013-03-11 18:58:19,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-3-thread-46) [1b011f22] SpmStopVDSCommand::Stopping SPM on vds cheetah01.scl.lab.tlv.redhat.com, pool id af0d05f3-f358-4df5-ac81-35684017b815 >2013-03-11 18:58:19,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-3-thread-46) [1b011f22] FINISH, SpmStopVDSCommand, log id: 626f7463 >2013-03-11 18:58:19,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (pool-3-thread-46) [1b011f22] FINISH, ResetIrsVDSCommand, log id: 7923050b >2013-03-11 18:58:19,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (pool-3-thread-46) [1b011f22] FINISH, SpmStopOnIrsVDSCommand, log id: 176d0fb8 >2013-03-11 18:58:19,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (pool-3-thread-46) [1b011f22] START, DisconnectStoragePoolVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = af0d05f3-f358-4df5-ac81-35684017b815, vds_spm_id = 1), log id: 669fce90 >2013-03-11 18:58:20,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (pool-3-thread-46) [1b011f22] FINISH, DisconnectStoragePoolVDSCommand, log id: 669fce90 >2013-03-11 18:58:20,008 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (pool-3-thread-46) [70987495] Running command: RemoveStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-11 18:58:20,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-3-thread-46) [70987495] START, DisconnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 1b2864b1-237c-4316-bff5-b7044c234a9d, connection: /data, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1e0bba85 >2013-03-11 18:58:22,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-3-thread-46) [70987495] FINISH, DisconnectStorageServerVDSCommand, return: {1b2864b1-237c-4316-bff5-b7044c234a9d=0}, log id: 1e0bba85 >2013-03-11 18:58:39,423 INFO [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (pool-3-thread-46) [370cee4c] Running command: RemoveStoragePoolCommand internal: false. Entities affected : ID: af0d05f3-f358-4df5-ac81-35684017b815 Type: StoragePool >2013-03-11 18:58:39,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-3-thread-45) [4cd4dc89] START, ValidateStorageServerConnectionVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 1b2864b1-237c-4316-bff5-b7044c234a9d, connection: /data, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 58f30fd6 >2013-03-11 18:58:39,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-3-thread-45) [4cd4dc89] FINISH, ValidateStorageServerConnectionVDSCommand, return: {1b2864b1-237c-4316-bff5-b7044c234a9d=0}, log id: 58f30fd6 >2013-03-11 18:58:39,508 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-3-thread-45) [4cd4dc89] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-11 18:58:39,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-3-thread-45) [4cd4dc89] START, ConnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 1b2864b1-237c-4316-bff5-b7044c234a9d, connection: /data, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 163936f1 >2013-03-11 18:58:39,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-3-thread-45) [4cd4dc89] FINISH, ConnectStorageServerVDSCommand, return: {1b2864b1-237c-4316-bff5-b7044c234a9d=0}, log id: 163936f1 >2013-03-11 18:58:39,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) [4cd4dc89] START, ConnectStoragePoolVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = af0d05f3-f358-4df5-ac81-35684017b815, vds_spm_id = 1, masterDomainId = 77c935fb-bd6e-4f85-84e0-2377bf89e0b8, masterVersion = 1), log id: 126e97d4 >2013-03-11 18:58:42,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) [4cd4dc89] FINISH, ConnectStoragePoolVDSCommand, log id: 126e97d4 >2013-03-11 18:58:42,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DestroyStoragePoolVDSCommand] (pool-3-thread-46) [370cee4c] START, DestroyStoragePoolVDSCommand( storagePoolId = af0d05f3-f358-4df5-ac81-35684017b815, ignoreFailoverLimit = false, compatabilityVersion = null), log id: 64dd5158 >2013-03-11 18:58:42,627 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (pool-3-thread-46) [66c105ff] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: af0d05f3-f358-4df5-ac81-35684017b815 Type: StoragePool >2013-03-11 18:58:42,654 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [66c105ff] hostFromVds::selectedVds - cheetah01.scl.lab.tlv.redhat.com, spmStatus Free, storage pool cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 18:58:42,657 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [66c105ff] starting spm on vds cheetah01.scl.lab.tlv.redhat.com, storage pool cheetah01-scl-lab-tlv-redhat-com-Local, prevId -1, LVER 0 >2013-03-11 18:58:42,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-3-thread-46) [66c105ff] START, SpmStartVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = af0d05f3-f358-4df5-ac81-35684017b815, prevId=-1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 44eb819b >2013-03-11 18:58:42,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-3-thread-46) [66c105ff] spmStart polling started: taskId = 385b2f45-8e34-43c3-b532-48b1bf199d69 >2013-03-11 18:58:43,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-3-thread-46) [66c105ff] spmStart polling ended: taskId = 385b2f45-8e34-43c3-b532-48b1bf199d69 task status = finished >2013-03-11 18:58:43,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-3-thread-46) [66c105ff] spmStart polling ended, spm status: SPM >2013-03-11 18:58:43,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-3-thread-46) [66c105ff] START, HSMClearTaskVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, taskId=385b2f45-8e34-43c3-b532-48b1bf199d69), log id: 49adc810 >2013-03-11 18:58:43,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-3-thread-46) [66c105ff] FINISH, HSMClearTaskVDSCommand, log id: 49adc810 >2013-03-11 18:58:43,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-3-thread-46) [66c105ff] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2fe008a, log id: 44eb819b >2013-03-11 18:58:43,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [66c105ff] Initialize Irs proxy from vds: cheetah01.scl.lab.tlv.redhat.com >2013-03-11 18:58:43,809 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-46) [66c105ff] IrsProxyData::disposing >2013-03-11 18:58:43,812 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DestroyStoragePoolVDSCommand] (pool-3-thread-46) [66c105ff] FINISH, DestroyStoragePoolVDSCommand, log id: 64dd5158 >2013-03-11 18:58:43,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (pool-3-thread-46) [66c105ff] START, FormatStorageDomainVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storageDomainId=77c935fb-bd6e-4f85-84e0-2377bf89e0b8), log id: 3e2d9cb4 >2013-03-11 18:58:46,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (pool-3-thread-46) [66c105ff] FINISH, FormatStorageDomainVDSCommand, log id: 3e2d9cb4 >2013-03-11 18:58:46,803 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (pool-3-thread-46) [464a6354] Running command: RemoveStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-11 18:58:46,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-3-thread-46) [464a6354] START, DisconnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 1b2864b1-237c-4316-bff5-b7044c234a9d, connection: /data, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6a05c291 >2013-03-11 18:58:49,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-3-thread-46) [464a6354] FINISH, DisconnectStorageServerVDSCommand, return: {1b2864b1-237c-4316-bff5-b7044c234a9d=0}, log id: 6a05c291 >2013-03-11 18:58:49,776 INFO [org.ovirt.engine.core.bll.storage.DisconnectStoragePoolAsyncOperation] (pool-3-thread-45) [464a6354] Disconnect storage pool treatment vds: cheetah01.scl.lab.tlv.redhat.com,pool cheetah01-scl-lab-tlv-redhat-com-Local >2013-03-11 18:58:49,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (pool-3-thread-45) [464a6354] START, DisconnectStoragePoolVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, storagePoolId = af0d05f3-f358-4df5-ac81-35684017b815, vds_spm_id = 1), log id: 7a31dad6 >2013-03-11 18:58:49,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (pool-3-thread-45) [464a6354] FINISH, DisconnectStoragePoolVDSCommand, log id: 7a31dad6 >2013-03-11 18:58:52,699 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-11 18:58:52,706 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:58:52,706 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 18:59:52,692 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-11 18:59:52,704 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 18:59:52,704 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [518e1018] Autorecovering 1 hosts >2013-03-11 19:00:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [518e1018] Autorecovering hosts id: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, name : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 19:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [325d87f7] Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 19:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [325d87f7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 19:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [325d87f7] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 602efe9b >2013-03-11 19:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-70) [325d87f7] FINISH, SetVdsStatusVDSCommand, log id: 602efe9b >2013-03-11 19:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-70) [325d87f7] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 6bcdb80f >2013-03-11 19:00:00,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [325d87f7] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 67351a3c >2013-03-11 19:00:00,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-70) [325d87f7] FINISH, GetHardwareInfoVDSCommand, log id: 67351a3c >2013-03-11 19:00:00,418 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-70) [729ab946] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 19:00:00,421 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-70) [727a26af] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 19:00:00,422 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (QuartzScheduler_Worker-70) [727a26af] FINISH, ActivateVdsVDSCommand, log id: 6bcdb80f >2013-03-11 19:00:00,441 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (QuartzScheduler_Worker-70) [727a26af] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 19:00:00,442 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-70) [727a26af] No string for VDS_ACTIVATE_ASYNC type. Use default Log >2013-03-11 19:00:00,444 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [727a26af] Autorecovering 1 storage domains >2013-03-11 19:00:00,445 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (QuartzScheduler_Worker-70) [727a26af] Autorecovering storage domains id: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 >2013-03-11 19:00:00,445 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [47bdcb14] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 19:00:00,446 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [47bdcb14] ConnectDomainToStorage. Before Connect all hosts to pool. Time:3/11/13 7:00 PM >2013-03-11 19:00:00,452 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (QuartzScheduler_Worker-70) [47bdcb14] ConnectDomainToStorage. After Connect all hosts to pool. Time:3/11/13 7:00 PM >2013-03-11 19:00:01,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [72fbc680] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 2c2256cc >2013-03-11 19:00:01,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-49) [72fbc680] FINISH, GetHardwareInfoVDSCommand, log id: 2c2256cc >2013-03-11 19:00:01,158 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-49) [6fa46145] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 19:00:01,161 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (QuartzScheduler_Worker-49) [e742a0f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 19:00:01,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) START, ConnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2, masterDomainId = 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126, masterVersion = 1), log id: 4cbadf12 >2013-03-11 19:00:08,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126']] >2013-03-11 19:00:08,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-3-thread-45) HostName = cheetah02.scl.lab.tlv.redhat.com >2013-03-11 19:00:08,204 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-45) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=5849b030-626e-47cb-ad90-3ce782d831b3, msdUUID=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126' >2013-03-11 19:00:08,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 4cbadf12 >2013-03-11 19:00:08,213 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-45) [91ed746] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 Type: Storage >2013-03-11 19:00:08,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [91ed746] START, MarkPoolInReconstructModeVDSCommand( storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 213bf1a1 >2013-03-11 19:00:08,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-45) [91ed746] clearing cache for problematic entities in pool 5849b030-626e-47cb-ad90-3ce782d831b3 >2013-03-11 19:00:08,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (pool-3-thread-45) [91ed746] FINISH, MarkPoolInReconstructModeVDSCommand, log id: 213bf1a1 >2013-03-11 19:00:08,287 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-3-thread-45) [91ed746] No string for RECONSTRUCT_MASTER_FAILED type. Use default Log >2013-03-11 19:00:08,287 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (pool-3-thread-46) [e742a0f] Finished reconstruct for pool 5849b030-626e-47cb-ad90-3ce782d831b3. Clearing event queue >2013-03-11 19:00:08,302 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (QuartzScheduler_Worker-49) [4b11437c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 19:00:08,304 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [4b11437c] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 6baff540 >2013-03-11 19:00:08,306 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (QuartzScheduler_Worker-49) [4b11437c] FINISH, SetVdsStatusVDSCommand, log id: 6baff540 >2013-03-11 19:00:08,312 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-49) [4b11437c] Try to add duplicate audit log values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value: storagepoolname >2013-03-11 19:00:08,349 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-49) [325d69ad] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 19:00:08,351 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-49) [ad31ea7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 19:00:08,353 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-49) [ad31ea7] Host 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com is already in NonOperational status. SetNonOperationalVds command is skipped. >2013-03-11 19:00:15,210 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-3-thread-46) [5428b904] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-11 19:00:15,212 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-46) [5428b904] START, SetVdsStatusVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 469eb145 >2013-03-11 19:00:15,215 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-46) [5428b904] FINISH, SetVdsStatusVDSCommand, log id: 469eb145 >2013-03-11 19:00:15,231 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-3-thread-46) [5428b904] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-11 19:00:18,049 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-20) [713599e7] Updated vds status from Preparing for Maintenance to Maintenance in database, vds = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 : cheetah01.scl.lab.tlv.redhat.com >2013-03-11 19:00:32,365 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-3-thread-46) [2b267677] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 19:00:32,367 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-46) [2b267677] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 5c0e39e6 >2013-03-11 19:00:32,370 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-46) [2b267677] FINISH, SetVdsStatusVDSCommand, log id: 5c0e39e6 >2013-03-11 19:00:32,390 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-3-thread-46) [2b267677] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 19:00:32,443 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-22) [39deeb81] Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 19:00:32,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-3-thread-50) Clearing cache of pool: 5849b030-626e-47cb-ad90-3ce782d831b3 for problematic entities of VDS: cheetah02.scl.lab.tlv.redhat.com. >2013-03-11 19:00:32,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (QuartzScheduler_Worker-22) [39deeb81] START, DisconnectStoragePoolVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, vds_spm_id = 2), log id: 52ae229d >2013-03-11 19:00:32,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (QuartzScheduler_Worker-22) [39deeb81] FINISH, DisconnectStoragePoolVDSCommand, log id: 52ae229d >2013-03-11 19:00:32,478 INFO [org.ovirt.engine.core.bll.storage.DisconnectHostFromStoragePoolServersCommand] (QuartzScheduler_Worker-22) [66e9fa4] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 19:00:32,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (QuartzScheduler_Worker-22) [66e9fa4] START, DisconnectStorageServerVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, storagePoolId = 5849b030-626e-47cb-ad90-3ce782d831b3, storageType = NFS, connectionList = []), log id: 5d6e3853 >2013-03-11 19:00:41,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (QuartzScheduler_Worker-22) [66e9fa4] FINISH, DisconnectStorageServerVDSCommand, return: {}, log id: 5d6e3853 >2013-03-11 19:00:50,388 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-10) [3af12644] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-11 19:00:52,696 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-11 19:00:52,704 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:00:52,706 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:01:22,317 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-6) [7642b08f] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: 2084b173-fc6d-477f-8273-81c080e9ac6d Type: StoragePool >2013-03-11 19:01:35,022 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp-/127.0.0.1:8702-6) [2e540311] Running command: ChangeVDSClusterCommand internal: false. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS, ID: 60101d93-4d90-4851-89a5-a0ad5185c26d Type: VdsGroups >2013-03-11 19:01:35,044 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-6) [2e540311] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 19:01:35,050 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-6) [2e540311] Lock freed to object EngineLock [exclusiveLocks= key: 2084b173-fc6d-477f-8273-81c080e9ac6d value: REGISTER_VDS >, sharedLocks= ] >2013-03-11 19:01:35,099 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-5) [2abe9ba4] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 19:01:38,195 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-6) Lock Acquired to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 19:01:38,209 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-3-thread-45) [aff580a] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 19:01:38,215 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-45) [aff580a] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=Unassigned, nonOperationalReason=NONE), log id: 3b2a5103 >2013-03-11 19:01:38,216 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-45) [aff580a] FINISH, SetVdsStatusVDSCommand, log id: 3b2a5103 >2013-03-11 19:01:38,219 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-3-thread-45) [aff580a] START, ActivateVdsVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12), log id: 47e21e8f >2013-03-11 19:01:38,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (pool-3-thread-45) [aff580a] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 3d1c1f4b >2013-03-11 19:01:38,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (pool-3-thread-45) [aff580a] FINISH, GetHardwareInfoVDSCommand, log id: 3d1c1f4b >2013-03-11 19:01:38,295 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-3-thread-45) [7bf39b06] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 19:01:38,298 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-3-thread-45) [2e54857d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 19:01:38,300 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-3-thread-45) [2e54857d] FINISH, ActivateVdsVDSCommand, log id: 47e21e8f >2013-03-11 19:01:38,319 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-3-thread-45) [2e54857d] Lock freed to object EngineLock [exclusiveLocks= key: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 value: VDS >, sharedLocks= ] >2013-03-11 19:01:38,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [9b9714b] START, GetHardwareInfoVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, vds=Host[cheetah02.scl.lab.tlv.redhat.com]), log id: 4174add2 >2013-03-11 19:01:38,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-28) [9b9714b] FINISH, GetHardwareInfoVDSCommand, log id: 4174add2 >2013-03-11 19:01:38,501 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-28) [8aa97b5] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 19:01:38,513 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-45) [8aa97b5] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2013-03-11 19:01:38,517 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-28) [14c4c533] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 19:01:38,521 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-28) [98b0baf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 19:01:45,528 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp-/127.0.0.1:8702-6) [4cadd2a6] Running command: ChangeVDSClusterCommand internal: false. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS, ID: 60101d93-4d90-4851-89a5-a0ad5185c26d Type: VdsGroups >2013-03-11 19:01:45,536 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-6) [4cadd2a6] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-11 19:01:45,541 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-6) [4cadd2a6] Lock freed to object EngineLock [exclusiveLocks= key: 2084b173-fc6d-477f-8273-81c080e9ac6d value: REGISTER_VDS >, sharedLocks= ] >2013-03-11 19:01:45,633 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-2) [246c9c6] Running command: UpdateVdsCommand internal: false. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-11 19:01:47,252 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-5) Lock Acquired to object EngineLock [exclusiveLocks= key: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 value: VDS >, sharedLocks= ] >2013-03-11 19:01:47,270 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-3-thread-45) [7b3140c8] Running command: ActivateVdsCommand internal: false. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-11 19:01:47,278 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-45) [7b3140c8] START, SetVdsStatusVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, status=Unassigned, nonOperationalReason=NONE), log id: 20a50ea3 >2013-03-11 19:01:47,279 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-45) [7b3140c8] FINISH, SetVdsStatusVDSCommand, log id: 20a50ea3 >2013-03-11 19:01:47,281 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-3-thread-45) [7b3140c8] START, ActivateVdsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0), log id: 76bdd050 >2013-03-11 19:01:47,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (pool-3-thread-45) [7b3140c8] START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: ea4a8e4 >2013-03-11 19:01:47,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (pool-3-thread-45) [7b3140c8] FINISH, GetHardwareInfoVDSCommand, log id: ea4a8e4 >2013-03-11 19:01:47,368 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-3-thread-45) [6344eec2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-11 19:01:47,370 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-3-thread-45) [359220d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-11 19:01:47,372 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-3-thread-45) [359220d] FINISH, ActivateVdsVDSCommand, log id: 76bdd050 >2013-03-11 19:01:47,396 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-3-thread-45) [359220d] Lock freed to object EngineLock [exclusiveLocks= key: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 value: VDS >, sharedLocks= ] >2013-03-11 19:01:48,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [fae2a73] START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: 2636056b >2013-03-11 19:01:48,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-66) [fae2a73] FINISH, GetHardwareInfoVDSCommand, log id: 2636056b >2013-03-11 19:01:48,223 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-66) [feb1d7e] Running command: InitVdsOnUpCommand internal: true. >2013-03-11 19:01:48,232 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-45) [feb1d7e] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2013-03-11 19:01:48,236 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-66) [2658b010] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-11 19:01:48,240 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-66) [11861f8f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-11 19:01:52,695 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-11 19:01:52,701 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:01:52,702 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:01:56,169 INFO [org.ovirt.engine.core.bll.storage.ForceRemoveStorageDomainCommand] (ajp-/127.0.0.1:8702-1) Lock Acquired to object EngineLock [exclusiveLocks= key: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 value: STORAGE >, sharedLocks= ] >2013-03-11 19:01:56,174 WARN [org.ovirt.engine.core.bll.storage.ForceRemoveStorageDomainCommand] (ajp-/127.0.0.1:8702-1) CanDoAction of action ForceRemoveStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DESTROY_DOMAIN,ERROR_CANNOT_DESTROY_LAST_STORAGE_DOMAIN >2013-03-11 19:01:56,174 INFO [org.ovirt.engine.core.bll.storage.ForceRemoveStorageDomainCommand] (ajp-/127.0.0.1:8702-1) Lock freed to object EngineLock [exclusiveLocks= key: 05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126 value: STORAGE >, sharedLocks= ] >2013-03-11 19:02:06,987 INFO [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (ajp-/127.0.0.1:8702-7) [1278e83d] Running command: RemoveStoragePoolCommand internal: false. Entities affected : ID: 5849b030-626e-47cb-ad90-3ce782d831b3 Type: StoragePool >2013-03-11 19:02:52,696 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 19:02:52,706 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:02:52,706 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:03:52,699 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-11 19:03:52,711 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:03:52,711 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:04:52,699 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-11 19:04:52,707 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:04:52,707 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:05:52,701 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 19:05:52,708 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:05:52,708 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:06:52,702 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 19:06:52,709 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:06:52,709 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:07:52,704 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-11 19:07:52,710 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:07:52,711 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:08:52,706 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-11 19:08:52,713 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:08:52,713 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:09:52,707 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-11 19:09:52,716 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:09:52,716 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:10:52,708 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-11 19:10:52,716 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:10:52,717 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:11:16,307 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-3-thread-45) [1971b60d] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 19:11:16,309 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-45) [1971b60d] START, SetVdsStatusVDSCommand(HostName = cheetah02.scl.lab.tlv.redhat.com, HostId = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 32e820d6 >2013-03-11 19:11:16,312 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-45) [1971b60d] FINISH, SetVdsStatusVDSCommand, log id: 32e820d6 >2013-03-11 19:11:16,326 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-3-thread-45) [1971b60d] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-11 19:11:17,830 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-35) [3c6eb9e5] Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 : cheetah02.scl.lab.tlv.redhat.com >2013-03-11 19:11:35,495 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-11 19:11:35,502 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:11:40,201 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-11 19:11:40,207 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:11:40,329 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-11 19:11:52,710 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-11 19:11:52,717 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:11:52,717 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:12:52,711 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-11 19:12:52,718 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:12:52,718 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:13:52,714 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-11 19:13:52,720 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:13:52,721 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:14:52,715 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-11 19:14:52,722 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:14:52,722 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:15:52,717 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-11 19:15:52,725 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:15:52,725 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:16:52,719 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-11 19:16:52,726 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:16:52,726 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:17:52,721 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-11 19:17:52,728 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:17:52,728 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:18:52,724 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-11 19:18:52,732 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:18:52,732 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:19:42,102 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-11 19:19:42,111 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:19:45,919 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-11 19:19:45,926 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:19:46,033 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-11 19:19:51,452 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-5) [74709505] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-11 19:19:52,730 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-11 19:19:52,755 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:19:52,756 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:19:53,553 WARN [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-1) [505c54b4] CanDoAction of action AddVdsGroup failed. Reasons:VAR__TYPE__CLUSTER,VAR__ACTION__CREATE,VDS_GROUP_CANNOT_DO_ACTION_NAME_IN_USE >2013-03-11 19:19:53,559 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8702-1) Operation Failed: [Cannot create Cluster. Cluster name is already in use.] >2013-03-11 19:20:52,726 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-11 19:20:52,735 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:20:52,735 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:21:37,295 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-3-thread-45) [5d4d3aac] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-11 19:21:37,297 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-45) [5d4d3aac] START, SetVdsStatusVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 1fa3bb1e >2013-03-11 19:21:37,301 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-45) [5d4d3aac] FINISH, SetVdsStatusVDSCommand, log id: 1fa3bb1e >2013-03-11 19:21:37,335 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-3-thread-45) [5d4d3aac] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-11 19:21:37,352 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-97) [44516e24] Updated vds status from Preparing for Maintenance to Maintenance in database, vds = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 : cheetah01.scl.lab.tlv.redhat.com >2013-03-11 19:21:52,727 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-11 19:21:52,734 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-11 19:21:52,734 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-11 19:22:04,166 INFO [org.ovirt.engine.core.bll.RemoveVdsGroupCommand] (pool-3-thread-45) [2ecd1cbe] Running command: RemoveVdsGroupCommand internal: false. Entities affected : ID: 0522bd09-2a55-4b55-95c9-2c8fb81d9382 Type: VdsGroups >2013-03-11 19:22:07,803 WARN [org.ovirt.engine.core.bll.RemoveVdsGroupCommand] (ajp-/127.0.0.1:8702-4) CanDoAction of action RemoveVdsGroup failed. Reasons:VAR__TYPE__CLUSTER,VAR__ACTION__REMOVE,VDS_CANNOT_REMOVE_VDS_GROUP_VDS_DETECTED >2013-03-11 19:30:33,902 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-53) [39386dd9] Start refreshing all users data >2013-03-11 19:30:35,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-40) [779cd84] Attempting to update VMs/Templates Ovf. >2013-03-11 20:30:33,904 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-79) [23b10d31] Start refreshing all users data >2013-03-11 20:30:35,141 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-67) [2b47e7da] Attempting to update VMs/Templates Ovf. >2013-03-11 21:30:33,907 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-52) [3b22dc02] Start refreshing all users data >2013-03-11 21:30:35,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-19) [16014079] Attempting to update VMs/Templates Ovf. >2013-03-11 22:30:33,909 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-3) [3864a1c4] Start refreshing all users data >2013-03-11 22:30:35,143 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-27) [3b7b20e9] Attempting to update VMs/Templates Ovf. >2013-03-11 23:30:33,911 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-13) [3df04d79] Start refreshing all users data >2013-03-11 23:30:35,144 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-78) [2312340a] Attempting to update VMs/Templates Ovf. >2013-03-12 00:30:33,913 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-34) [13cac3c6] Start refreshing all users data >2013-03-12 00:30:35,145 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-72) [133e5ad0] Attempting to update VMs/Templates Ovf. >2013-03-12 01:30:33,915 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-89) [1044299d] Start refreshing all users data >2013-03-12 01:30:35,146 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-59) [3d4e1575] Attempting to update VMs/Templates Ovf. >2013-03-12 02:30:33,917 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-2) [5ba55ac3] Start refreshing all users data >2013-03-12 02:30:35,147 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-96) [4e36b1a] Attempting to update VMs/Templates Ovf. >2013-03-12 03:30:33,919 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-11) [291c51f9] Start refreshing all users data >2013-03-12 03:30:35,148 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [66a03382] Attempting to update VMs/Templates Ovf. >2013-03-12 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (QuartzScheduler_Worker-57) [af1b9f7] AuditLogCleanupManager::deleteAgedOutAuditLogs - entered >2013-03-12 04:30:33,921 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-70) [47bdcb14] Start refreshing all users data >2013-03-12 04:30:35,150 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-88) [46dff337] Attempting to update VMs/Templates Ovf. >2013-03-12 05:30:33,923 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-64) [24d55122] Start refreshing all users data >2013-03-12 05:30:35,151 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-20) [713599e7] Attempting to update VMs/Templates Ovf. >2013-03-12 06:30:33,925 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-50) [7c3cf50c] Start refreshing all users data >2013-03-12 06:30:35,152 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-73) [4b624214] Attempting to update VMs/Templates Ovf. >2013-03-12 07:30:33,928 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-35) [3c6eb9e5] Start refreshing all users data >2013-03-12 07:30:35,154 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-57) [af1b9f7] Attempting to update VMs/Templates Ovf. >2013-03-12 08:30:33,929 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-69) [26bcedab] Start refreshing all users data >2013-03-12 08:30:35,155 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-93) [39f376f4] Attempting to update VMs/Templates Ovf. >2013-03-12 09:30:33,931 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-11) [291c51f9] Start refreshing all users data >2013-03-12 09:30:35,156 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-12) [66a03382] Attempting to update VMs/Templates Ovf. >2013-03-12 10:30:33,934 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-51) [1b748202] Start refreshing all users data >2013-03-12 10:30:35,157 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-16) [38fc60d6] Attempting to update VMs/Templates Ovf. >2013-03-12 11:30:33,936 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-49) [ad31ea7] Start refreshing all users data >2013-03-12 11:30:35,159 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-78) [2312340a] Attempting to update VMs/Templates Ovf. >2013-03-12 12:30:33,938 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-50) [7c3cf50c] Start refreshing all users data >2013-03-12 12:30:35,160 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-98) [500d17ae] Attempting to update VMs/Templates Ovf. >2013-03-12 13:30:33,940 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-10) [7b712e38] Start refreshing all users data >2013-03-12 13:30:35,161 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-35) [3c6eb9e5] Attempting to update VMs/Templates Ovf. >2013-03-12 14:15:15,958 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginAdminUserCommand internal: false. >2013-03-12 14:15:16,011 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 14:15:16,017 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:15:16,018 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:15:17,792 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (WebAdminUpperRightButtonURL) with null version, using default general for version >2013-03-12 14:16:16,092 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 14:16:16,099 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:16:16,100 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:17:16,086 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 14:17:16,094 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:17:16,094 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:18:16,088 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 14:18:16,093 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:18:16,093 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:19:16,087 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 14:19:16,094 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:19:16,094 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:20:16,097 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 14:20:16,112 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:20:16,112 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:21:16,093 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 14:21:16,099 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:21:16,099 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:22:16,094 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 14:22:16,102 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:22:16,102 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:22:41,478 INFO [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (pool-3-thread-45) [53e5a9ee] Running command: RemoveStoragePoolCommand internal: false. Entities affected : ID: 0362dae2-af8e-4e1b-93fa-529f075621e7 Type: StoragePool >2013-03-12 14:23:01,269 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp-/127.0.0.1:8702-1) [7493fba8] Running command: ChangeVDSClusterCommand internal: false. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS, ID: 6a3926cd-1ef9-4303-8cb9-0842e0ee44e7 Type: VdsGroups >2013-03-12 14:23:01,285 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-1) [7493fba8] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-12 14:23:01,291 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-1) [7493fba8] Lock freed to object EngineLock [exclusiveLocks= key: da9b3f49-2f13-482b-9b6a-55da6dd7dbfb value: REGISTER_VDS >, sharedLocks= ] >2013-03-12 14:23:01,367 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-2) [63c9a0ec] Running command: UpdateVdsCommand internal: false. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-12 14:23:02,304 ERROR [org.ovirt.engine.core.bll.GetoVirtISOsQuery] (ajp-/127.0.0.1:8702-11) ovirt ISOs directory not found. Search in: /usr/share/ovirt-node-iso >2013-03-12 14:23:16,095 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 14:23:16,102 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:23:16,102 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:23:26,180 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp-/127.0.0.1:8702-11) [3cfd037f] Running command: ChangeVDSClusterCommand internal: false. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS, ID: 6a3926cd-1ef9-4303-8cb9-0842e0ee44e7 Type: VdsGroups >2013-03-12 14:23:26,193 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-11) [3cfd037f] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-12 14:23:26,197 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-11) [3cfd037f] Lock freed to object EngineLock [exclusiveLocks= key: da9b3f49-2f13-482b-9b6a-55da6dd7dbfb value: REGISTER_VDS >, sharedLocks= ] >2013-03-12 14:23:26,271 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-10) [4e50d38] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-12 14:23:31,928 INFO [org.ovirt.engine.core.bll.RemoveVdsGroupCommand] (pool-3-thread-45) [180b7254] Running command: RemoveVdsGroupCommand internal: false. Entities affected : ID: 60101d93-4d90-4851-89a5-a0ad5185c26d Type: VdsGroups >2013-03-12 14:23:41,204 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-6) Lock Acquired to object EngineLock [exclusiveLocks= key: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 value: VDS >, sharedLocks= ] >2013-03-12 14:23:41,213 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-3-thread-45) [1845cf67] Running command: RemoveVdsCommand internal: false. Entities affected : ID: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 Type: VDS >2013-03-12 14:23:41,279 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-3-thread-45) [1845cf67] START, RemoveVdsVDSCommand( HostId = cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0), log id: ce4a3a3 >2013-03-12 14:23:41,280 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-3-thread-45) [1845cf67] vdsManager::disposing >2013-03-12 14:23:41,280 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-3-thread-45) [1845cf67] FINISH, RemoveVdsVDSCommand, log id: ce4a3a3 >2013-03-12 14:23:41,281 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-3-thread-45) [1845cf67] Lock freed to object EngineLock [exclusiveLocks= key: cd108f1e-c1f2-4eda-9a2e-b1d79503e7d0 value: VDS >, sharedLocks= ] >2013-03-12 14:24:01,587 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-10) [41986cea] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: 2084b173-fc6d-477f-8273-81c080e9ac6d Type: StoragePool >2013-03-12 14:24:09,690 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp-/127.0.0.1:8702-7) [6aef26ac] Running command: ChangeVDSClusterCommand internal: false. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS, ID: b02b6b77-6fd7-444d-a14b-46facd8132d6 Type: VdsGroups >2013-03-12 14:24:09,701 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-7) [6aef26ac] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-12 14:24:09,705 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-7) [6aef26ac] Lock freed to object EngineLock [exclusiveLocks= key: 2084b173-fc6d-477f-8273-81c080e9ac6d value: REGISTER_VDS >, sharedLocks= ] >2013-03-12 14:24:09,775 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-11) [69fbb87a] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 5e9ad544-fe9f-4669-8b11-1631ba7d1c12 Type: VDS >2013-03-12 14:24:16,105 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 14:24:16,112 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:24:16,112 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:24:29,175 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 14:24:29,180 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:24:34,103 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 14:24:34,110 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:24:34,233 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-12 14:24:41,607 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-11) [4ed0cb45] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-12 14:24:41,897 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-5) [3eee24fd] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: 3f096c1a-c2dc-43ff-a26d-c2afea3d7b0a Type: StoragePool >2013-03-12 14:24:43,716 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-6) [3138c5dc] Running command: AddVdsCommand internal: false. Entities affected : ID: 4b432b40-ca13-4f0b-80df-a4b642d00a53 Type: VdsGroups >2013-03-12 14:24:43,730 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-6) [67ed1f22] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 6eb7cb1f-50ff-40be-8165-7b8270323231 Type: VDS >2013-03-12 14:24:43,734 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-6) [67ed1f22] Lock freed to object EngineLock [exclusiveLocks= key: 3f096c1a-c2dc-43ff-a26d-c2afea3d7b0a value: REGISTER_VDS >, sharedLocks= ] >2013-03-12 14:24:43,734 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-6) [67ed1f22] Cannot get vdsManager for vdsid=6eb7cb1f-50ff-40be-8165-7b8270323231 >2013-03-12 14:24:43,740 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [67ed1f22] START, RemoveVdsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 6eb7cb1f-50ff-40be-8165-7b8270323231), log id: 30261599 >2013-03-12 14:24:43,741 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-6) [67ed1f22] Cannot get vdsManager for vdsid=6eb7cb1f-50ff-40be-8165-7b8270323231 >2013-03-12 14:24:43,741 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [67ed1f22] FINISH, RemoveVdsVDSCommand, log id: 30261599 >2013-03-12 14:24:43,741 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-6) [67ed1f22] Cannot get vdsManager for vdsid=6eb7cb1f-50ff-40be-8165-7b8270323231 >2013-03-12 14:24:43,742 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [67ed1f22] START, AddVdsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 6eb7cb1f-50ff-40be-8165-7b8270323231), log id: 445df753 >2013-03-12 14:24:43,742 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [67ed1f22] AddVds - entered , starting logic to add VDS 6eb7cb1f-50ff-40be-8165-7b8270323231 >2013-03-12 14:24:43,743 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [67ed1f22] AddVds - VDS 6eb7cb1f-50ff-40be-8165-7b8270323231 was added, will try to add it to the resource manager >2013-03-12 14:24:43,744 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-6) [67ed1f22] Eneterd VdsManager:constructor >2013-03-12 14:24:43,744 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-6) [67ed1f22] vdsBroker(10.35.160.152,54,321) >2013-03-12 14:24:43,745 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-6) [67ed1f22] ResourceManager::AddVds - VDS 6eb7cb1f-50ff-40be-8165-7b8270323231 was added to the Resource Manager >2013-03-12 14:24:43,745 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [67ed1f22] FINISH, AddVdsVDSCommand, log id: 445df753 >2013-03-12 14:24:43,766 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-45) [7f408ccd] Running command: InstallVdsCommand internal: true. Entities affected : ID: 6eb7cb1f-50ff-40be-8165-7b8270323231 Type: VDS >2013-03-12 14:24:43,766 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-45) [7f408ccd] Before Installation host 6eb7cb1f-50ff-40be-8165-7b8270323231, cheetah01.scl.lab.tlv.redhat.com >2013-03-12 14:24:43,848 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-3-thread-45) [7f408ccd] Installation 10.35.160.152: Connected to host 10.35.160.152 with SSH key fingerprint: 13:fc:73:f1:4a:7c:f5:6a:13:3c:b9:e2:50:10:73:7f >2013-03-12 14:24:43,862 INFO [org.ovirt.engine.core.bll.VdsDeploy] (pool-3-thread-45) [7f408ccd] Installation of 10.35.160.152. Executing command via SSH umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2013-03-12 14:24:43,870 INFO [org.ovirt.engine.core.utils.ssh.SSHDialog] (pool-3-thread-45) SSH execute root@10.35.160.152 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2013-03-12 14:24:44,665 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Initializing >2013-03-12 14:24:44,679 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Environment setup >2013-03-12 14:24:44,730 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Installation packages setup >2013-03-12 14:24:46,747 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-27) [3b7b20e9] Initializing Host: cheetah01.scl.lab.tlv.redhat.com >2013-03-12 14:25:14,824 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Yum Downloading: epel/primary_db 2.8 M(56%) >2013-03-12 14:25:16,099 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 14:25:16,106 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:25:16,107 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:25:42,749 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Programs detection >2013-03-12 14:25:42,754 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Environment customization >2013-03-12 14:25:42,764 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Logs at host located at: '/tmp/ovirt-host-deploy-20130312142446.log' >2013-03-12 14:25:42,789 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host 10.35.160.152 reports unique id 75f723b4-e40e-4955-b7f1-ab62144e9278 >2013-03-12 14:25:42,803 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 75f723b4-e40e-4955-b7f1-ab62144e9278 to Host 10.35.160.152 >2013-03-12 14:25:42,832 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Enfocing host reboot >2013-03-12 14:25:42,840 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Setup validation >2013-03-12 14:25:42,871 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Hardware supports virtualization >2013-03-12 14:25:42,877 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Transaction setup >2013-03-12 14:25:42,881 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Package installation >2013-03-12 14:25:44,200 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Misc configuration >2013-03-12 14:25:44,225 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Setting time >2013-03-12 14:25:45,924 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Enrolling certificate >2013-03-12 14:25:47,199 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Transaction commit >2013-03-12 14:25:47,204 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Closing up >2013-03-12 14:25:47,546 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Reboot scheduled >2013-03-12 14:25:47,550 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Pre-termination >2013-03-12 14:25:47,563 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Reboot scheduled >2013-03-12 14:25:47,569 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20130312142547-10.35.160.152-7f408ccd.log' >2013-03-12 14:25:47,700 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Termination >2013-03-12 14:25:47,825 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-45) [7f408ccd] After Installation host cheetah01.scl.lab.tlv.redhat.com, VDS >2013-03-12 14:25:47,825 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-50) [7f408ccd] Waiting 300 seconds, for server to finish reboot process. >2013-03-12 14:26:16,103 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 14:26:16,109 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:26:16,109 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:27:16,105 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 14:27:16,114 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:27:16,114 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:28:16,104 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 14:28:16,112 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:28:16,112 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:29:16,107 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 14:29:16,114 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:29:16,114 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:30:16,129 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 14:30:16,136 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:30:16,137 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:30:33,942 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-15) [2e714cfa] Start refreshing all users data >2013-03-12 14:30:35,163 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-32) [56ec593d] Attempting to update VMs/Templates Ovf. >2013-03-12 14:30:47,828 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-50) [7f408ccd] START, SetVdsStatusVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 6eb7cb1f-50ff-40be-8165-7b8270323231, status=NonResponsive, nonOperationalReason=NONE), log id: a5d5ab8 >2013-03-12 14:30:47,831 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-50) [7f408ccd] FINISH, SetVdsStatusVDSCommand, log id: a5d5ab8 >2013-03-12 14:31:16,110 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 14:31:16,115 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:31:16,115 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:32:16,113 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 14:32:16,120 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:32:16,120 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:33:16,114 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 14:33:16,121 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:33:16,121 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:34:16,115 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 14:34:16,122 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:34:16,122 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:35:16,118 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 14:35:16,126 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:35:16,126 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:36:16,119 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 14:36:16,126 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:36:16,126 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:37:16,120 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 14:37:16,127 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:37:16,128 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:38:16,128 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 14:38:16,137 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:38:16,137 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:39:16,124 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 14:39:16,130 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:39:16,131 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:40:16,126 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 14:40:16,134 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:40:16,134 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:41:16,127 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 14:41:16,135 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:41:16,135 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:42:16,129 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 14:42:16,136 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:42:16,136 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:43:16,130 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 14:43:16,137 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:43:16,137 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:43:34,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [eb68cea] START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 6eb7cb1f-50ff-40be-8165-7b8270323231, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: 5f8d5913 >2013-03-12 14:43:35,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [eb68cea] FINISH, GetHardwareInfoVDSCommand, log id: 5f8d5913 >2013-03-12 14:43:35,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [eb68cea] START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 6eb7cb1f-50ff-40be-8165-7b8270323231, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: 799709a0 >2013-03-12 14:43:35,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-80) [eb68cea] FINISH, GetHardwareInfoVDSCommand, log id: 799709a0 >2013-03-12 14:43:35,185 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-80) [6cfa65d8] Running command: InitVdsOnUpCommand internal: true. >2013-03-12 14:43:35,195 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-50) [6cfa65d8] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2013-03-12 14:43:35,203 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-80) [2d8517b5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 6eb7cb1f-50ff-40be-8165-7b8270323231 Type: VDS >2013-03-12 14:43:35,209 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-80) [77c0f644] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 6eb7cb1f-50ff-40be-8165-7b8270323231 Type: VDS >2013-03-12 14:44:16,132 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 14:44:16,140 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:44:16,140 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:45:16,134 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 14:45:16,142 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:45:16,142 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:46:16,136 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 14:46:16,141 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:46:16,142 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:47:16,142 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 14:47:16,149 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:47:16,150 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:48:16,140 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 14:48:16,147 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:48:16,147 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:49:16,142 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 14:49:16,150 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:49:16,150 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:50:16,143 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 14:50:16,154 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:50:16,155 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:51:16,142 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 14:51:16,150 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:51:16,150 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:52:16,146 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 14:52:16,154 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:52:16,154 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:53:16,148 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 14:53:16,157 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:53:16,157 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:54:16,150 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 14:54:16,165 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:54:16,165 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:55:10,240 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-3-thread-45) [5be9c8a] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 6eb7cb1f-50ff-40be-8165-7b8270323231 Type: VDS >2013-03-12 14:55:10,242 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-45) [5be9c8a] START, SetVdsStatusVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 6eb7cb1f-50ff-40be-8165-7b8270323231, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 9224a1f >2013-03-12 14:55:10,246 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-45) [5be9c8a] FINISH, SetVdsStatusVDSCommand, log id: 9224a1f >2013-03-12 14:55:10,275 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-3-thread-45) [5be9c8a] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 6eb7cb1f-50ff-40be-8165-7b8270323231 Type: VDS >2013-03-12 14:55:10,396 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-41) [3d4a3bd8] Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 6eb7cb1f-50ff-40be-8165-7b8270323231 : cheetah01.scl.lab.tlv.redhat.com >2013-03-12 14:55:16,152 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 14:55:16,160 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:55:16,160 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:55:23,032 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-2) Lock Acquired to object EngineLock [exclusiveLocks= key: 6eb7cb1f-50ff-40be-8165-7b8270323231 value: VDS >, sharedLocks= ] >2013-03-12 14:55:23,038 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-3-thread-45) [5c2d998d] Running command: RemoveVdsCommand internal: false. Entities affected : ID: 6eb7cb1f-50ff-40be-8165-7b8270323231 Type: VDS >2013-03-12 14:55:23,063 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-3-thread-45) [5c2d998d] START, RemoveVdsVDSCommand( HostId = 6eb7cb1f-50ff-40be-8165-7b8270323231), log id: 64df78f9 >2013-03-12 14:55:23,063 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-3-thread-45) [5c2d998d] vdsManager::disposing >2013-03-12 14:55:23,063 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-3-thread-45) [5c2d998d] FINISH, RemoveVdsVDSCommand, log id: 64df78f9 >2013-03-12 14:55:23,064 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-3-thread-45) [5c2d998d] Lock freed to object EngineLock [exclusiveLocks= key: 6eb7cb1f-50ff-40be-8165-7b8270323231 value: VDS >, sharedLocks= ] >2013-03-12 14:55:31,562 INFO [org.ovirt.engine.core.bll.RemoveVdsGroupCommand] (pool-3-thread-45) [7ddb4d68] Running command: RemoveVdsGroupCommand internal: false. Entities affected : ID: 4b432b40-ca13-4f0b-80df-a4b642d00a53 Type: VdsGroups >2013-03-12 14:55:36,044 INFO [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (pool-3-thread-45) [1dc63508] Running command: RemoveStoragePoolCommand internal: false. Entities affected : ID: 3f096c1a-c2dc-43ff-a26d-c2afea3d7b0a Type: StoragePool >2013-03-12 14:55:47,498 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 14:55:47,505 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:55:51,714 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 14:55:51,721 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:55:51,834 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-12 14:55:56,734 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-11) [5b7fc8aa] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-12 14:55:57,053 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-5) [118afdad] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: 94f7de7a-824a-469c-8a08-bba26fb5650f Type: StoragePool >2013-03-12 14:55:58,152 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-6) [218704c6] Running command: AddVdsCommand internal: false. Entities affected : ID: 5ea7c2a5-31bf-4abb-b054-87a716b4f6b6 Type: VdsGroups >2013-03-12 14:55:58,168 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-6) [41302d87] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: b5f408a3-27c2-4492-8393-e1ab42dd1c5b Type: VDS >2013-03-12 14:55:58,172 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-6) [41302d87] Lock freed to object EngineLock [exclusiveLocks= key: 94f7de7a-824a-469c-8a08-bba26fb5650f value: REGISTER_VDS >, sharedLocks= ] >2013-03-12 14:55:58,172 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-6) [41302d87] Cannot get vdsManager for vdsid=b5f408a3-27c2-4492-8393-e1ab42dd1c5b >2013-03-12 14:55:58,175 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [41302d87] START, RemoveVdsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = b5f408a3-27c2-4492-8393-e1ab42dd1c5b), log id: 560e8a3c >2013-03-12 14:55:58,175 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-6) [41302d87] Cannot get vdsManager for vdsid=b5f408a3-27c2-4492-8393-e1ab42dd1c5b >2013-03-12 14:55:58,175 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [41302d87] FINISH, RemoveVdsVDSCommand, log id: 560e8a3c >2013-03-12 14:55:58,175 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-6) [41302d87] Cannot get vdsManager for vdsid=b5f408a3-27c2-4492-8393-e1ab42dd1c5b >2013-03-12 14:55:58,182 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [41302d87] START, AddVdsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = b5f408a3-27c2-4492-8393-e1ab42dd1c5b), log id: 7d52e0e2 >2013-03-12 14:55:58,182 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [41302d87] AddVds - entered , starting logic to add VDS b5f408a3-27c2-4492-8393-e1ab42dd1c5b >2013-03-12 14:55:58,183 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [41302d87] AddVds - VDS b5f408a3-27c2-4492-8393-e1ab42dd1c5b was added, will try to add it to the resource manager >2013-03-12 14:55:58,184 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-6) [41302d87] Eneterd VdsManager:constructor >2013-03-12 14:55:58,184 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-6) [41302d87] vdsBroker(10.35.160.152,54,321) >2013-03-12 14:55:58,185 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-6) [41302d87] ResourceManager::AddVds - VDS b5f408a3-27c2-4492-8393-e1ab42dd1c5b was added to the Resource Manager >2013-03-12 14:55:58,185 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-6) [41302d87] FINISH, AddVdsVDSCommand, log id: 7d52e0e2 >2013-03-12 14:55:58,216 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-45) [9469d21] Running command: InstallVdsCommand internal: true. Entities affected : ID: b5f408a3-27c2-4492-8393-e1ab42dd1c5b Type: VDS >2013-03-12 14:55:58,217 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-45) [9469d21] Before Installation host b5f408a3-27c2-4492-8393-e1ab42dd1c5b, cheetah01.scl.lab.tlv.redhat.com >2013-03-12 14:55:58,294 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-3-thread-45) [9469d21] Installation 10.35.160.152: Connected to host 10.35.160.152 with SSH key fingerprint: 13:fc:73:f1:4a:7c:f5:6a:13:3c:b9:e2:50:10:73:7f >2013-03-12 14:55:58,308 INFO [org.ovirt.engine.core.bll.VdsDeploy] (pool-3-thread-45) [9469d21] Installation of 10.35.160.152. Executing command via SSH umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2013-03-12 14:55:58,314 INFO [org.ovirt.engine.core.utils.ssh.SSHDialog] (pool-3-thread-45) SSH execute root@10.35.160.152 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2013-03-12 14:55:59,118 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Initializing >2013-03-12 14:55:59,129 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Environment setup >2013-03-12 14:55:59,187 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Installation packages setup >2013-03-12 14:56:01,188 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-13) [3df04d79] Initializing Host: cheetah01.scl.lab.tlv.redhat.com >2013-03-12 14:56:04,033 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Programs detection >2013-03-12 14:56:04,039 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Environment customization >2013-03-12 14:56:04,056 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Logs at host located at: '/tmp/ovirt-host-deploy-20130312145601.log' >2013-03-12 14:56:04,088 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host 10.35.160.152 reports unique id 75f723b4-e40e-4955-b7f1-ab62144e9278 >2013-03-12 14:56:04,111 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 75f723b4-e40e-4955-b7f1-ab62144e9278 to Host 10.35.160.152 >2013-03-12 14:56:04,142 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Enfocing host reboot >2013-03-12 14:56:04,152 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Setup validation >2013-03-12 14:56:04,304 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Hardware supports virtualization >2013-03-12 14:56:04,308 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Transaction setup >2013-03-12 14:56:04,313 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Package installation >2013-03-12 14:56:05,633 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Misc configuration >2013-03-12 14:56:05,654 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Setting time >2013-03-12 14:56:08,296 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Enrolling certificate >2013-03-12 14:56:09,567 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Transaction commit >2013-03-12 14:56:09,573 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Closing up >2013-03-12 14:56:09,920 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Reboot scheduled >2013-03-12 14:56:09,924 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Pre-termination >2013-03-12 14:56:09,942 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Reboot scheduled >2013-03-12 14:56:09,947 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20130312145609-10.35.160.152-9469d21.log' >2013-03-12 14:56:10,071 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Termination >2013-03-12 14:56:10,218 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-45) [9469d21] After Installation host cheetah01.scl.lab.tlv.redhat.com, VDS >2013-03-12 14:56:10,218 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-50) [9469d21] Waiting 300 seconds, for server to finish reboot process. >2013-03-12 14:56:16,154 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 14:56:16,163 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:56:16,163 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:57:16,157 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 14:57:16,164 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:57:16,164 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:58:16,158 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 14:58:16,166 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:58:16,166 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 14:59:16,160 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 14:59:16,168 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 14:59:16,168 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:00:16,163 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 15:00:16,170 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:00:16,170 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:00:47,043 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginAdminUserCommand internal: false. >2013-03-12 15:00:47,184 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 15:00:47,191 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:00:47,191 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:00:48,892 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (WebAdminUpperRightButtonURL) with null version, using default general for version >2013-03-12 15:01:10,221 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-50) [9469d21] START, SetVdsStatusVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = b5f408a3-27c2-4492-8393-e1ab42dd1c5b, status=NonResponsive, nonOperationalReason=NONE), log id: 222eeec0 >2013-03-12 15:01:10,224 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-50) [9469d21] FINISH, SetVdsStatusVDSCommand, log id: 222eeec0 >2013-03-12 15:01:16,163 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 15:01:16,168 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:01:16,169 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:01:47,302 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 15:01:47,310 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:01:47,310 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:02:16,165 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 15:02:16,172 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:02:16,172 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:02:47,303 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 15:02:47,311 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:02:47,311 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:03:16,167 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 15:03:16,174 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:03:16,175 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:03:47,318 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 15:03:47,327 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:03:47,327 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:04:16,169 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 15:04:16,176 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:04:16,176 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:04:47,308 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 15:04:47,315 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:04:47,315 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:05:16,171 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 15:05:16,178 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:05:16,178 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:05:47,308 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 15:05:47,316 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:05:47,316 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:06:16,173 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 15:06:16,180 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:06:16,180 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:06:47,316 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 15:06:47,321 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:06:47,322 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:07:16,174 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 15:07:16,181 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:07:16,181 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:07:47,309 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 15:07:47,316 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:07:47,316 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:08:16,176 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 15:08:16,183 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:08:16,183 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:08:47,317 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 15:08:47,325 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:08:47,325 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:09:16,177 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 15:09:16,186 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:09:16,186 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:09:47,312 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 15:09:47,320 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:09:47,320 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:10:16,179 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 15:10:16,186 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:10:16,186 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:10:47,316 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 15:10:47,323 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:10:47,324 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:11:16,181 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 15:11:16,189 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:11:16,189 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:11:47,315 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 15:11:47,323 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:11:47,323 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:12:00,640 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-3-thread-50) [7c1a6857] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: b5f408a3-27c2-4492-8393-e1ab42dd1c5b Type: VDS >2013-03-12 15:12:00,657 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-3-thread-50) [7c1a6857] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: b5f408a3-27c2-4492-8393-e1ab42dd1c5b Type: VDS >2013-03-12 15:12:00,660 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-50) [7c1a6857] START, SetVdsStatusVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = b5f408a3-27c2-4492-8393-e1ab42dd1c5b, status=Maintenance, nonOperationalReason=NONE), log id: 308a44b1 >2013-03-12 15:12:00,661 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-50) [7c1a6857] FINISH, SetVdsStatusVDSCommand, log id: 308a44b1 >2013-03-12 15:12:16,183 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 15:12:16,190 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:12:16,190 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:12:27,365 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-10) [67875396] Running command: UpdateVdsCommand internal: false. Entities affected : ID: b5f408a3-27c2-4492-8393-e1ab42dd1c5b Type: VDS >2013-03-12 15:12:27,376 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-10) [67875396] START, SetVdsStatusVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = b5f408a3-27c2-4492-8393-e1ab42dd1c5b, status=Installing, nonOperationalReason=NONE), log id: 46f211c3 >2013-03-12 15:12:27,379 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-10) [67875396] FINISH, SetVdsStatusVDSCommand, log id: 46f211c3 >2013-03-12 15:12:27,382 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-50) [42eeb3ec] Running command: InstallVdsCommand internal: true. Entities affected : ID: b5f408a3-27c2-4492-8393-e1ab42dd1c5b Type: VDS >2013-03-12 15:12:27,382 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-50) [42eeb3ec] Before Installation host b5f408a3-27c2-4492-8393-e1ab42dd1c5b, cheetah01.scl.lab.tlv.redhat.com >2013-03-12 15:12:27,456 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-3-thread-50) [42eeb3ec] Installation 10.35.160.152: Connected to host 10.35.160.152 with SSH key fingerprint: 13:fc:73:f1:4a:7c:f5:6a:13:3c:b9:e2:50:10:73:7f >2013-03-12 15:12:27,470 INFO [org.ovirt.engine.core.bll.VdsDeploy] (pool-3-thread-50) [42eeb3ec] Installation of 10.35.160.152. Executing command via SSH umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2013-03-12 15:12:27,478 INFO [org.ovirt.engine.core.utils.ssh.SSHDialog] (pool-3-thread-50) SSH execute root@10.35.160.152 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2013-03-12 15:12:28,300 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Initializing >2013-03-12 15:12:28,312 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Environment setup >2013-03-12 15:12:28,371 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Installation packages setup >2013-03-12 15:12:35,274 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Programs detection >2013-03-12 15:12:35,278 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Environment customization >2013-03-12 15:12:35,288 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Logs at host located at: '/tmp/ovirt-host-deploy-20130312151230.log' >2013-03-12 15:12:35,318 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host 10.35.160.152 reports unique id 75f723b4-e40e-4955-b7f1-ab62144e9278 >2013-03-12 15:12:35,343 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 75f723b4-e40e-4955-b7f1-ab62144e9278 to Host 10.35.160.152 >2013-03-12 15:12:35,367 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Enfocing host reboot >2013-03-12 15:12:35,375 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Setup validation >2013-03-12 15:12:35,502 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Hardware supports virtualization >2013-03-12 15:12:35,507 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Transaction setup >2013-03-12 15:12:35,511 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Package installation >2013-03-12 15:12:36,843 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Misc configuration >2013-03-12 15:12:36,925 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Setting time >2013-03-12 15:12:38,521 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Enrolling certificate >2013-03-12 15:12:39,806 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Transaction commit >2013-03-12 15:12:39,818 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Closing up >2013-03-12 15:12:40,464 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Reboot scheduled >2013-03-12 15:12:40,469 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Pre-termination >2013-03-12 15:12:40,479 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Reboot scheduled >2013-03-12 15:12:40,484 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20130312151240-10.35.160.152-42eeb3ec.log' >2013-03-12 15:12:40,615 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Termination >2013-03-12 15:12:40,734 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-50) [42eeb3ec] After Installation host cheetah01.scl.lab.tlv.redhat.com, VDS >2013-03-12 15:12:40,734 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-45) [42eeb3ec] Waiting 300 seconds, for server to finish reboot process. >2013-03-12 15:12:47,318 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 15:12:47,325 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:12:47,325 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:13:16,183 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 15:13:16,190 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:13:16,191 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:13:47,323 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 15:13:47,328 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:13:47,328 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:14:16,185 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 15:14:16,193 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:14:16,193 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:14:47,323 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 15:14:47,330 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:14:47,330 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:15:16,186 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 15:15:16,194 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:15:16,194 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:15:47,346 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 15:15:47,354 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:15:47,354 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:16:16,189 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 15:16:16,195 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:16:16,196 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:16:47,322 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 15:16:47,330 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:16:47,330 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:17:16,190 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 15:17:16,198 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:17:16,198 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:17:40,736 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-45) [42eeb3ec] START, SetVdsStatusVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = b5f408a3-27c2-4492-8393-e1ab42dd1c5b, status=NonResponsive, nonOperationalReason=NONE), log id: 723518a3 >2013-03-12 15:17:40,740 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-45) [42eeb3ec] FINISH, SetVdsStatusVDSCommand, log id: 723518a3 >2013-03-12 15:17:44,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [2e714cfa] START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = b5f408a3-27c2-4492-8393-e1ab42dd1c5b, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: 3c533912 >2013-03-12 15:17:44,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [2e714cfa] FINISH, GetHardwareInfoVDSCommand, log id: 3c533912 >2013-03-12 15:17:44,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [2e714cfa] START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = b5f408a3-27c2-4492-8393-e1ab42dd1c5b, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: 201f118b >2013-03-12 15:17:44,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-15) [2e714cfa] FINISH, GetHardwareInfoVDSCommand, log id: 201f118b >2013-03-12 15:17:44,305 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-15) [1ca4cd9f] Running command: InitVdsOnUpCommand internal: true. >2013-03-12 15:17:44,316 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-45) [1ca4cd9f] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2013-03-12 15:17:44,320 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-15) [158f842f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b5f408a3-27c2-4492-8393-e1ab42dd1c5b Type: VDS >2013-03-12 15:17:44,326 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-15) [57aa08b7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b5f408a3-27c2-4492-8393-e1ab42dd1c5b Type: VDS >2013-03-12 15:18:16,192 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 15:18:16,200 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:18:16,200 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:18:24,440 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-3-thread-45) [440d5f5c] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: b5f408a3-27c2-4492-8393-e1ab42dd1c5b Type: VDS >2013-03-12 15:18:24,442 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-45) [440d5f5c] START, SetVdsStatusVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = b5f408a3-27c2-4492-8393-e1ab42dd1c5b, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 793a4bef >2013-03-12 15:18:24,445 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-45) [440d5f5c] FINISH, SetVdsStatusVDSCommand, log id: 793a4bef >2013-03-12 15:18:24,461 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-3-thread-45) [440d5f5c] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: b5f408a3-27c2-4492-8393-e1ab42dd1c5b Type: VDS >2013-03-12 15:18:26,820 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-29) [5ea98a1a] Updated vds status from Preparing for Maintenance to Maintenance in database, vds = b5f408a3-27c2-4492-8393-e1ab42dd1c5b : cheetah01.scl.lab.tlv.redhat.com >2013-03-12 15:18:30,141 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-3) Lock Acquired to object EngineLock [exclusiveLocks= key: b5f408a3-27c2-4492-8393-e1ab42dd1c5b value: VDS >, sharedLocks= ] >2013-03-12 15:18:30,148 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-3-thread-45) [5df398da] Running command: RemoveVdsCommand internal: false. Entities affected : ID: b5f408a3-27c2-4492-8393-e1ab42dd1c5b Type: VDS >2013-03-12 15:18:30,178 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-3-thread-45) [5df398da] START, RemoveVdsVDSCommand( HostId = b5f408a3-27c2-4492-8393-e1ab42dd1c5b), log id: 744ac71a >2013-03-12 15:18:30,178 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-3-thread-45) [5df398da] vdsManager::disposing >2013-03-12 15:18:30,178 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-3-thread-45) [5df398da] FINISH, RemoveVdsVDSCommand, log id: 744ac71a >2013-03-12 15:18:30,179 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-3-thread-45) [5df398da] Lock freed to object EngineLock [exclusiveLocks= key: b5f408a3-27c2-4492-8393-e1ab42dd1c5b value: VDS >, sharedLocks= ] >2013-03-12 15:18:35,154 INFO [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (pool-3-thread-45) [5fcac7fe] Running command: RemoveStoragePoolCommand internal: false. Entities affected : ID: 94f7de7a-824a-469c-8a08-bba26fb5650f Type: StoragePool >2013-03-12 15:18:39,970 INFO [org.ovirt.engine.core.bll.RemoveVdsGroupCommand] (pool-3-thread-45) [70f2ab1] Running command: RemoveVdsGroupCommand internal: false. Entities affected : ID: 5ea7c2a5-31bf-4abb-b054-87a716b4f6b6 Type: VdsGroups >2013-03-12 15:18:52,448 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 15:18:52,455 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:18:56,463 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 15:18:56,470 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:18:56,584 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-12 15:19:01,368 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-5) [6100c03a] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-12 15:19:01,664 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-10) [21e4f29] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: 4c48df0d-5b6d-4a8a-b75a-f7292db3efbb Type: StoragePool >2013-03-12 15:19:02,648 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-5) [58b3bdee] Running command: AddVdsCommand internal: false. Entities affected : ID: 3e40d4e9-4b92-4dc5-9814-8c75cc0fe815 Type: VdsGroups >2013-03-12 15:19:02,660 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-5) [10c9be5c] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 8344d82f-72e9-4ec9-a04a-d39883abe78a Type: VDS >2013-03-12 15:19:02,664 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-5) [10c9be5c] Lock freed to object EngineLock [exclusiveLocks= key: 4c48df0d-5b6d-4a8a-b75a-f7292db3efbb value: REGISTER_VDS >, sharedLocks= ] >2013-03-12 15:19:02,664 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-5) [10c9be5c] Cannot get vdsManager for vdsid=8344d82f-72e9-4ec9-a04a-d39883abe78a >2013-03-12 15:19:02,666 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [10c9be5c] START, RemoveVdsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 8344d82f-72e9-4ec9-a04a-d39883abe78a), log id: 71ece286 >2013-03-12 15:19:02,666 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-5) [10c9be5c] Cannot get vdsManager for vdsid=8344d82f-72e9-4ec9-a04a-d39883abe78a >2013-03-12 15:19:02,666 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [10c9be5c] FINISH, RemoveVdsVDSCommand, log id: 71ece286 >2013-03-12 15:19:02,666 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-5) [10c9be5c] Cannot get vdsManager for vdsid=8344d82f-72e9-4ec9-a04a-d39883abe78a >2013-03-12 15:19:02,667 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [10c9be5c] START, AddVdsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 8344d82f-72e9-4ec9-a04a-d39883abe78a), log id: 4a0d0011 >2013-03-12 15:19:02,668 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [10c9be5c] AddVds - entered , starting logic to add VDS 8344d82f-72e9-4ec9-a04a-d39883abe78a >2013-03-12 15:19:02,669 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [10c9be5c] AddVds - VDS 8344d82f-72e9-4ec9-a04a-d39883abe78a was added, will try to add it to the resource manager >2013-03-12 15:19:02,669 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-5) [10c9be5c] Eneterd VdsManager:constructor >2013-03-12 15:19:02,669 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-5) [10c9be5c] vdsBroker(10.35.160.152,54,321) >2013-03-12 15:19:02,670 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-5) [10c9be5c] ResourceManager::AddVds - VDS 8344d82f-72e9-4ec9-a04a-d39883abe78a was added to the Resource Manager >2013-03-12 15:19:02,670 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [10c9be5c] FINISH, AddVdsVDSCommand, log id: 4a0d0011 >2013-03-12 15:19:02,693 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-45) [175bf05] Running command: InstallVdsCommand internal: true. Entities affected : ID: 8344d82f-72e9-4ec9-a04a-d39883abe78a Type: VDS >2013-03-12 15:19:02,693 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-45) [175bf05] Before Installation host 8344d82f-72e9-4ec9-a04a-d39883abe78a, cheetah01.scl.lab.tlv.redhat.com >2013-03-12 15:19:02,754 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-3-thread-45) [175bf05] Installation 10.35.160.152: Connected to host 10.35.160.152 with SSH key fingerprint: 13:fc:73:f1:4a:7c:f5:6a:13:3c:b9:e2:50:10:73:7f >2013-03-12 15:19:02,769 INFO [org.ovirt.engine.core.bll.VdsDeploy] (pool-3-thread-45) [175bf05] Installation of 10.35.160.152. Executing command via SSH umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2013-03-12 15:19:02,774 INFO [org.ovirt.engine.core.utils.ssh.SSHDialog] (pool-3-thread-45) SSH execute root@10.35.160.152 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2013-03-12 15:19:03,566 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Initializing >2013-03-12 15:19:03,577 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Environment setup >2013-03-12 15:19:03,624 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Installation packages setup >2013-03-12 15:19:05,672 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-43) [4be55521] Initializing Host: cheetah01.scl.lab.tlv.redhat.com >2013-03-12 15:19:07,719 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Programs detection >2013-03-12 15:19:07,725 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Environment customization >2013-03-12 15:19:07,733 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Logs at host located at: '/tmp/ovirt-host-deploy-20130312151905.log' >2013-03-12 15:19:07,760 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host 10.35.160.152 reports unique id 75f723b4-e40e-4955-b7f1-ab62144e9278 >2013-03-12 15:19:07,775 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 75f723b4-e40e-4955-b7f1-ab62144e9278 to Host 10.35.160.152 >2013-03-12 15:19:07,800 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Enfocing host reboot >2013-03-12 15:19:07,809 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Setup validation >2013-03-12 15:19:08,265 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Hardware supports virtualization >2013-03-12 15:19:08,275 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Transaction setup >2013-03-12 15:19:08,283 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Package installation >2013-03-12 15:19:09,598 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Misc configuration >2013-03-12 15:19:09,625 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Setting time >2013-03-12 15:19:12,225 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Enrolling certificate >2013-03-12 15:19:13,487 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Transaction commit >2013-03-12 15:19:13,492 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Closing up >2013-03-12 15:19:13,848 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Reboot scheduled >2013-03-12 15:19:13,853 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Pre-termination >2013-03-12 15:19:13,864 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Reboot scheduled >2013-03-12 15:19:13,868 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20130312151913-10.35.160.152-175bf05.log' >2013-03-12 15:19:13,988 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Termination >2013-03-12 15:19:14,100 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-45) [175bf05] After Installation host cheetah01.scl.lab.tlv.redhat.com, VDS >2013-03-12 15:19:14,100 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-50) [175bf05] Waiting 300 seconds, for server to finish reboot process. >2013-03-12 15:19:16,193 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 15:19:16,200 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:19:16,200 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:20:16,196 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 15:20:16,207 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:20:16,207 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:21:16,197 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 15:21:16,205 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:21:16,205 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:22:16,198 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 15:22:16,206 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:22:16,207 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:23:16,201 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 15:23:16,208 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:23:16,208 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:24:14,102 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-50) [175bf05] START, SetVdsStatusVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 8344d82f-72e9-4ec9-a04a-d39883abe78a, status=NonResponsive, nonOperationalReason=NONE), log id: 667c4511 >2013-03-12 15:24:14,106 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-50) [175bf05] FINISH, SetVdsStatusVDSCommand, log id: 667c4511 >2013-03-12 15:24:15,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [753f9c8a] START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 8344d82f-72e9-4ec9-a04a-d39883abe78a, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: 7d901a32 >2013-03-12 15:24:15,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [753f9c8a] FINISH, GetHardwareInfoVDSCommand, log id: 7d901a32 >2013-03-12 15:24:15,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [753f9c8a] START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 8344d82f-72e9-4ec9-a04a-d39883abe78a, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: 3401eecc >2013-03-12 15:24:15,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-94) [753f9c8a] FINISH, GetHardwareInfoVDSCommand, log id: 3401eecc >2013-03-12 15:24:15,445 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-94) [392bb08] Running command: InitVdsOnUpCommand internal: true. >2013-03-12 15:24:15,457 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-50) [392bb08] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2013-03-12 15:24:15,459 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-94) [7e999926] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 8344d82f-72e9-4ec9-a04a-d39883abe78a Type: VDS >2013-03-12 15:24:15,464 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-94) [66feaebf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 8344d82f-72e9-4ec9-a04a-d39883abe78a Type: VDS >2013-03-12 15:24:16,202 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 15:24:16,208 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:24:16,208 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:25:16,204 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 15:25:16,212 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:25:16,213 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:26:16,205 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 15:26:16,213 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:26:16,213 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:27:16,208 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 15:27:16,215 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:27:16,215 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:28:16,209 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 15:28:16,216 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:28:16,216 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:29:16,211 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 15:29:16,218 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:29:16,218 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:30:16,213 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 15:30:16,221 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:30:16,221 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:30:33,944 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-34) [13cac3c6] Start refreshing all users data >2013-03-12 15:30:35,164 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-77) [cf61334] Attempting to update VMs/Templates Ovf. >2013-03-12 15:31:16,216 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 15:31:16,222 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:31:16,222 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:32:16,215 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 15:32:16,223 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:32:16,223 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:33:16,218 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 15:33:16,225 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:33:16,226 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:34:16,219 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 15:34:16,227 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:34:16,227 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:35:16,221 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 15:35:16,228 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:35:16,228 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:36:16,223 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 15:36:16,230 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:36:16,231 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:37:16,225 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 15:37:16,232 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:37:16,232 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:38:16,228 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 15:38:16,236 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:38:16,237 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:39:16,227 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 15:39:16,235 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:39:16,235 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:40:16,231 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 15:40:16,238 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:40:16,238 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:40:33,358 INFO [org.ovirt.engine.core.bll.MaintananceNumberOfVdssCommand] (pool-3-thread-45) [46270266] Running command: MaintananceNumberOfVdssCommand internal: false. Entities affected : ID: 8344d82f-72e9-4ec9-a04a-d39883abe78a Type: VDS >2013-03-12 15:40:33,360 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-45) [46270266] START, SetVdsStatusVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 8344d82f-72e9-4ec9-a04a-d39883abe78a, status=PreparingForMaintenance, nonOperationalReason=NONE), log id: 37931570 >2013-03-12 15:40:33,363 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-45) [46270266] FINISH, SetVdsStatusVDSCommand, log id: 37931570 >2013-03-12 15:40:33,379 INFO [org.ovirt.engine.core.bll.MaintananceVdsCommand] (pool-3-thread-45) [46270266] Running command: MaintananceVdsCommand internal: true. Entities affected : ID: 8344d82f-72e9-4ec9-a04a-d39883abe78a Type: VDS >2013-03-12 15:40:36,279 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-42) [69cf856a] Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 8344d82f-72e9-4ec9-a04a-d39883abe78a : cheetah01.scl.lab.tlv.redhat.com >2013-03-12 15:40:46,741 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (ajp-/127.0.0.1:8702-9) Lock Acquired to object EngineLock [exclusiveLocks= key: 8344d82f-72e9-4ec9-a04a-d39883abe78a value: VDS >, sharedLocks= ] >2013-03-12 15:40:46,747 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-3-thread-45) [7a8b9efc] Running command: RemoveVdsCommand internal: false. Entities affected : ID: 8344d82f-72e9-4ec9-a04a-d39883abe78a Type: VDS >2013-03-12 15:40:46,778 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-3-thread-45) [7a8b9efc] START, RemoveVdsVDSCommand( HostId = 8344d82f-72e9-4ec9-a04a-d39883abe78a), log id: 5fd90e5e >2013-03-12 15:40:46,778 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-3-thread-45) [7a8b9efc] vdsManager::disposing >2013-03-12 15:40:46,778 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (pool-3-thread-45) [7a8b9efc] FINISH, RemoveVdsVDSCommand, log id: 5fd90e5e >2013-03-12 15:40:46,779 INFO [org.ovirt.engine.core.bll.RemoveVdsCommand] (pool-3-thread-45) [7a8b9efc] Lock freed to object EngineLock [exclusiveLocks= key: 8344d82f-72e9-4ec9-a04a-d39883abe78a value: VDS >, sharedLocks= ] >2013-03-12 15:41:16,232 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 15:41:16,239 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:41:16,240 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:41:24,620 INFO [org.ovirt.engine.core.bll.RemoveVdsGroupCommand] (pool-3-thread-45) [6b5f397b] Running command: RemoveVdsGroupCommand internal: false. Entities affected : ID: 3e40d4e9-4b92-4dc5-9814-8c75cc0fe815 Type: VdsGroups >2013-03-12 15:41:29,574 INFO [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (pool-3-thread-45) [66c51f7a] Running command: RemoveStoragePoolCommand internal: false. Entities affected : ID: 4c48df0d-5b6d-4a8a-b75a-f7292db3efbb Type: StoragePool >2013-03-12 15:41:38,783 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 15:41:38,791 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:41:45,577 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 15:41:45,584 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:41:45,702 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-12 15:41:50,636 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-10) [16ce6c7e] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-12 15:41:50,959 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-4) [82fbc36] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: 61a1cd37-ed09-4b5e-9217-e5e132875b08 Type: StoragePool >2013-03-12 15:41:51,954 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-10) [438baabb] Running command: AddVdsCommand internal: false. Entities affected : ID: 9a8f96d0-7a5b-4c5a-a0c0-5ed7c8786bf0 Type: VdsGroups >2013-03-12 15:41:51,966 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-10) [4d4b1625] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1 Type: VDS >2013-03-12 15:41:51,979 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-10) [4d4b1625] Lock freed to object EngineLock [exclusiveLocks= key: 61a1cd37-ed09-4b5e-9217-e5e132875b08 value: REGISTER_VDS >, sharedLocks= ] >2013-03-12 15:41:51,979 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-10) [4d4b1625] Cannot get vdsManager for vdsid=7ca749d9-3a07-4d55-ab0e-43bb37fafcc1 >2013-03-12 15:41:51,992 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [4d4b1625] START, RemoveVdsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1), log id: 44284c9e >2013-03-12 15:41:51,992 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-10) [4d4b1625] Cannot get vdsManager for vdsid=7ca749d9-3a07-4d55-ab0e-43bb37fafcc1 >2013-03-12 15:41:51,992 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [4d4b1625] FINISH, RemoveVdsVDSCommand, log id: 44284c9e >2013-03-12 15:41:51,992 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-10) [4d4b1625] Cannot get vdsManager for vdsid=7ca749d9-3a07-4d55-ab0e-43bb37fafcc1 >2013-03-12 15:41:51,993 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [4d4b1625] START, AddVdsVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1), log id: 335ade95 >2013-03-12 15:41:51,994 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [4d4b1625] AddVds - entered , starting logic to add VDS 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1 >2013-03-12 15:41:51,995 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [4d4b1625] AddVds - VDS 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1 was added, will try to add it to the resource manager >2013-03-12 15:41:51,996 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-10) [4d4b1625] Eneterd VdsManager:constructor >2013-03-12 15:41:51,996 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-10) [4d4b1625] vdsBroker(10.35.160.152,54,321) >2013-03-12 15:41:51,996 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-10) [4d4b1625] ResourceManager::AddVds - VDS 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1 was added to the Resource Manager >2013-03-12 15:41:51,996 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-10) [4d4b1625] FINISH, AddVdsVDSCommand, log id: 335ade95 >2013-03-12 15:41:52,021 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-45) [2f6cb1c] Running command: InstallVdsCommand internal: true. Entities affected : ID: 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1 Type: VDS >2013-03-12 15:41:52,021 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-45) [2f6cb1c] Before Installation host 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1, cheetah01.scl.lab.tlv.redhat.com >2013-03-12 15:41:52,084 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-3-thread-45) [2f6cb1c] Installation 10.35.160.152: Connected to host 10.35.160.152 with SSH key fingerprint: 13:fc:73:f1:4a:7c:f5:6a:13:3c:b9:e2:50:10:73:7f >2013-03-12 15:41:52,097 INFO [org.ovirt.engine.core.bll.VdsDeploy] (pool-3-thread-45) [2f6cb1c] Installation of 10.35.160.152. Executing command via SSH umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2013-03-12 15:41:52,103 INFO [org.ovirt.engine.core.utils.ssh.SSHDialog] (pool-3-thread-45) SSH execute root@10.35.160.152 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2013-03-12 15:41:52,898 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Initializing >2013-03-12 15:41:52,909 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Environment setup >2013-03-12 15:41:52,963 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Installation packages setup >2013-03-12 15:41:54,998 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (QuartzScheduler_Worker-2) [5ba55ac3] Initializing Host: cheetah01.scl.lab.tlv.redhat.com >2013-03-12 15:41:55,283 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Programs detection >2013-03-12 15:41:55,287 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Environment customization >2013-03-12 15:41:55,297 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Logs at host located at: '/tmp/ovirt-host-deploy-20130312154154.log' >2013-03-12 15:41:55,320 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host 10.35.160.152 reports unique id 75f723b4-e40e-4955-b7f1-ab62144e9278 >2013-03-12 15:41:55,335 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 75f723b4-e40e-4955-b7f1-ab62144e9278 to Host 10.35.160.152 >2013-03-12 15:41:55,376 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Enfocing host reboot >2013-03-12 15:41:55,384 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Setup validation >2013-03-12 15:41:55,522 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Hardware supports virtualization >2013-03-12 15:41:55,527 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Transaction setup >2013-03-12 15:41:55,531 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Package installation >2013-03-12 15:41:56,853 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Misc configuration >2013-03-12 15:41:56,872 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Setting time >2013-03-12 15:41:58,471 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Enrolling certificate >2013-03-12 15:41:59,722 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Transaction commit >2013-03-12 15:41:59,727 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Closing up >2013-03-12 15:42:00,081 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Reboot scheduled >2013-03-12 15:42:00,086 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Pre-termination >2013-03-12 15:42:00,096 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Reboot scheduled >2013-03-12 15:42:00,100 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20130312154200-10.35.160.152-2f6cb1c.log' >2013-03-12 15:42:00,221 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.35.160.152: Stage: Termination >2013-03-12 15:42:00,347 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-45) [2f6cb1c] After Installation host cheetah01.scl.lab.tlv.redhat.com, VDS >2013-03-12 15:42:00,347 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-3-thread-50) [2f6cb1c] Waiting 300 seconds, for server to finish reboot process. >2013-03-12 15:42:16,240 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 15:42:16,247 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:42:16,247 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:43:16,236 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 15:43:16,244 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:43:16,244 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:44:16,237 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 15:44:16,245 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:44:16,245 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:45:16,240 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 15:45:16,255 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:45:16,256 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:46:16,241 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 15:46:16,248 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:46:16,248 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:47:00,349 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-50) [2f6cb1c] START, SetVdsStatusVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1, status=NonResponsive, nonOperationalReason=NONE), log id: 42972d7e >2013-03-12 15:47:00,353 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-3-thread-50) [2f6cb1c] FINISH, SetVdsStatusVDSCommand, log id: 42972d7e >2013-03-12 15:47:01,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [1044299d] START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: 236ee8c3 >2013-03-12 15:47:01,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [1044299d] FINISH, GetHardwareInfoVDSCommand, log id: 236ee8c3 >2013-03-12 15:47:01,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [1044299d] START, GetHardwareInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1, vds=Host[cheetah01.scl.lab.tlv.redhat.com]), log id: 7ba07117 >2013-03-12 15:47:01,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (QuartzScheduler_Worker-89) [1044299d] FINISH, GetHardwareInfoVDSCommand, log id: 7ba07117 >2013-03-12 15:47:01,736 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (QuartzScheduler_Worker-89) [33d4531] Running command: InitVdsOnUpCommand internal: true. >2013-03-12 15:47:01,749 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (pool-3-thread-50) [33d4531] MultipleActionsRunner of type MigrateVmToServer invoked with no actions >2013-03-12 15:47:01,751 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (QuartzScheduler_Worker-89) [7918612f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1 Type: VDS >2013-03-12 15:47:01,756 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (QuartzScheduler_Worker-89) [524d5d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1 Type: VDS >2013-03-12 15:47:16,243 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 15:47:16,252 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:47:16,252 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:48:16,245 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 15:48:16,253 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:48:16,253 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:49:16,246 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 15:49:16,253 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:49:16,253 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:50:16,247 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 15:50:16,263 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:50:16,264 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:51:16,249 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 15:51:16,256 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:51:16,256 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:52:16,251 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 15:52:16,258 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:52:16,259 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:53:16,252 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 15:53:16,259 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:53:16,259 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:54:16,253 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 15:54:16,260 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:54:16,260 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:55:16,254 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 15:55:16,264 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:55:16,264 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:56:16,275 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 15:56:16,283 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:56:16,283 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:57:16,258 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 15:57:16,265 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:57:16,266 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:58:16,260 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 15:58:16,267 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:58:16,267 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 15:59:16,262 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 15:59:16,269 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 15:59:16,269 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:00:16,264 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 16:00:16,272 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:00:16,273 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:00:48,021 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 16:00:48,029 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:00:51,944 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 16:00:51,953 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:00:52,063 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-12 16:01:16,265 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:01:16,272 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:01:16,273 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:02:16,267 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 16:02:16,274 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:02:16,274 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:02:38,881 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 16:02:38,886 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:02:48,410 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 16:02:48,417 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:02:48,538 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-12 16:03:16,268 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 16:03:16,276 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:03:16,276 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:04:16,270 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 16:04:16,277 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:04:16,277 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:05:16,272 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:05:16,281 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:05:16,281 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:06:06,133 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 16:06:06,140 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:06:15,124 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 16:06:15,131 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:06:15,242 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-12 16:06:16,273 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 16:06:16,280 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:06:16,281 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:06:28,064 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:06:28,072 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:06:28,681 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 16:06:28,690 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:06:29,064 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 16:06:29,072 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:06:29,123 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:06:29,130 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:06:29,524 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:06:29,533 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:06:29,943 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-1) [4b8e260] Lock Acquired to object EngineLock [exclusiveLocks= key: rhel_6_3_shortVM value: VM_NAME >, sharedLocks= ] >2013-03-12 16:06:29,992 ERROR [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-1) [4b8e260] Error during CanDoActionFailure.: java.lang.NullPointerException > at org.ovirt.engine.core.bll.AddVmCommand.fillImagesMapBasedOnTemplate(AddVmCommand.java:414) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVmCommand.fillDestMap(AddVmCommand.java:409) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVmCommand.buildAndCheckDestStorageDomains(AddVmCommand.java:360) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVmCommand.canDoAction(AddVmCommand.java:297) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVmFromScratchCommand.canDoAction(AddVmFromScratchCommand.java:140) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.internalCanDoAction(CommandBase.java:658) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:311) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:361) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.RunAction(Backend.java:307) [engine-bll.jar:] > at sun.reflect.GeneratedMethodAccessor165.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.ovirt.engine.core.utils.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:11) [engine-utils.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:209) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:361) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:192) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:181) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view8.RunAction(Unknown Source) [engine-common.jar:] > at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:191) > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:249) > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:115) > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:105) > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:124) > at org.ovirt.engine.api.restapi.resource.BackendVmsResource.addVmFromScratch(BackendVmsResource.java:297) > at org.ovirt.engine.api.restapi.resource.BackendVmsResource.add(BackendVmsResource.java:118) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:536) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:513) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:125) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec.jar:1.0.1.Final-redhat-2] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:372) > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:453) > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:931) > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2013-03-12 16:06:29,999 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-1) [4b8e260] Lock freed to object EngineLock [exclusiveLocks= key: rhel_6_3_shortVM value: VM_NAME >, sharedLocks= ] >2013-03-12 16:06:30,002 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8702-1) Operation Failed: [General command validation failure.] >2013-03-12 16:06:30,063 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 16:06:30,070 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:07:16,275 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 16:07:16,283 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:07:16,283 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:08:16,277 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:08:16,284 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:08:16,284 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:09:16,278 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 16:09:16,286 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:09:16,286 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:10:16,280 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 16:10:16,295 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:10:16,295 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:10:56,268 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 16:10:56,275 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:11:00,381 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 16:11:00,387 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:11:00,490 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (SupportedClusterLevels) with null version, using default general for version >2013-03-12 16:11:06,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-5) [5dc61f3b] START, ValidateStorageServerConnectionVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 390f0553 >2013-03-12 16:11:07,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-5) [5dc61f3b] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 390f0553 >2013-03-12 16:11:07,015 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-5) [5dc61f3b] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-12 16:11:07,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [5dc61f3b] START, ConnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1e0de259 >2013-03-12 16:11:07,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [5dc61f3b] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 1e0de259 >2013-03-12 16:11:07,860 INFO [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery] (ajp-/127.0.0.1:8702-5) START, GetExistingStorageDomainListQuery(org.ovirt.engine.core.common.queries.GetExistingStorageDomainListParameters@7abc7253), log id: 52423d25 >2013-03-12 16:11:07,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp-/127.0.0.1:8702-5) START, HSMGetStorageDomainsListVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1, storagePoolId=00000000-0000-0000-0000-000000000000, storageType=NFS, storageDomainType=Data, path=lion.qa.lab.tlv.redhat.com:/fastpass/gickowic/nfs1), log id: 22e2d9b2 >2013-03-12 16:11:16,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp-/127.0.0.1:8702-5) FINISH, HSMGetStorageDomainsListVDSCommand, return: [05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126], log id: 22e2d9b2 >2013-03-12 16:11:16,241 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp-/127.0.0.1:8702-5) START, HSMGetStorageDomainInfoVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1, storageDomainId=05e9f330-fc2f-4ef4-8f1b-36c6e6fc1126), log id: 5767429b >2013-03-12 16:11:16,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp-/127.0.0.1:8702-5) FINISH, HSMGetStorageDomainInfoVDSCommand, return: org.ovirt.engine.core.common.utils.Pair@de142d60, log id: 5767429b >2013-03-12 16:11:16,271 INFO [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery] (ajp-/127.0.0.1:8702-5) FINISH, GetExistingStorageDomainListQuery, log id: 52423d25 >2013-03-12 16:11:16,282 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 16:11:16,291 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:11:16,291 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:11:16,301 WARN [org.ovirt.engine.core.bll.storage.AddExistingNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-5) [78cf530] CanDoAction of action AddExistingNFSStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ADD,ACTION_TYPE_FAILED_IMPORT_DATA_DOMAIN_PROHIBITED >2013-03-12 16:11:16,305 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8702-5) Operation Failed: [Importing data domain is prohibited.] >2013-03-12 16:11:16,522 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:11:16,529 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:11:16,685 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 16:11:16,692 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:11:16,784 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 16:11:16,792 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:11:16,845 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 16:11:16,856 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:11:16,925 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 16:11:16,933 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:11:16,966 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-3) [77d41134] Lock Acquired to object EngineLock [exclusiveLocks= key: rhel_6_3_shortVM value: VM_NAME >, sharedLocks= ] >2013-03-12 16:11:16,975 ERROR [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-3) [77d41134] Error during CanDoActionFailure.: java.lang.NullPointerException > at org.ovirt.engine.core.bll.AddVmCommand.fillImagesMapBasedOnTemplate(AddVmCommand.java:414) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVmCommand.fillDestMap(AddVmCommand.java:409) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVmCommand.buildAndCheckDestStorageDomains(AddVmCommand.java:360) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVmCommand.canDoAction(AddVmCommand.java:297) [engine-bll.jar:] > at org.ovirt.engine.core.bll.AddVmFromScratchCommand.canDoAction(AddVmFromScratchCommand.java:140) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.internalCanDoAction(CommandBase.java:658) [engine-bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:311) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:361) [engine-bll.jar:] > at org.ovirt.engine.core.bll.Backend.RunAction(Backend.java:307) [engine-bll.jar:] > at sun.reflect.GeneratedMethodAccessor165.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.ovirt.engine.core.utils.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:11) [engine-utils.jar:] > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) [:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:209) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:361) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:192) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:181) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.1.Final-redhat-2] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.1.3.Final-redhat-4] > at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view8.RunAction(Unknown Source) [engine-common.jar:] > at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:191) > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:249) > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:115) > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:105) > at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:124) > at org.ovirt.engine.api.restapi.resource.BackendVmsResource.addVmFromScratch(BackendVmsResource.java:297) > at org.ovirt.engine.api.restapi.resource.BackendVmsResource.add(BackendVmsResource.java:118) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_09-icedtea] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_09-icedtea] > at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_09-icedtea] > at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:536) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:513) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:125) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs.jar:2.3.4.Final-redhat-2] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec.jar:1.0.1.Final-redhat-2] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:372) > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:453) > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:931) > at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea] > >2013-03-12 16:11:16,980 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-3) [77d41134] Lock freed to object EngineLock [exclusiveLocks= key: rhel_6_3_shortVM value: VM_NAME >, sharedLocks= ] >2013-03-12 16:11:16,982 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8702-3) Operation Failed: [General command validation failure.] >2013-03-12 16:11:17,033 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 16:11:17,041 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:12:16,284 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:12:16,297 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:12:16,298 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:13:16,286 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 16:13:16,294 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:13:16,294 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:14:16,286 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:14:16,294 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:14:16,294 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:15:16,288 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 16:15:16,296 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:15:16,297 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:16:16,289 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:16:16,298 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:16:16,298 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:17:16,292 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 16:17:16,299 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:17:16,299 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:18:16,293 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 16:18:16,300 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:18:16,301 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:19:16,294 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 16:19:16,299 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:19:16,300 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:20:16,296 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 16:20:16,306 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:20:16,306 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:20:24,809 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginAdminUserCommand internal: false. >2013-03-12 16:20:24,851 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 16:20:24,864 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:20:24,864 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:20:25,787 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (WebAdminUpperRightButtonURL) with null version, using default general for version >2013-03-12 16:21:16,297 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 16:21:16,303 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:21:16,303 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:21:17,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-3) [67fdc793] START, ValidateStorageServerConnectionVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: dbf92ee >2013-03-12 16:21:17,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (ajp-/127.0.0.1:8702-3) [67fdc793] FINISH, ValidateStorageServerConnectionVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: dbf92ee >2013-03-12 16:21:17,961 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-3) [67fdc793] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-12 16:21:17,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-3) [67fdc793] START, ConnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: ed373ff0-f77d-4b77-94db-4bc33fbe74e5, connection: lion.qa.lab.tlv.redhat.com:/fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5195be2d >2013-03-12 16:21:17,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-3) [67fdc793] FINISH, ConnectStorageServerVDSCommand, return: {ed373ff0-f77d-4b77-94db-4bc33fbe74e5=0}, log id: 5195be2d >2013-03-12 16:21:18,031 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [6405acd7] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-12 16:21:18,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [6405acd7] START, CreateStorageDomainVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1, storageDomain=org.ovirt.engine.core.common.businessentities.StorageDomainStatic@9cdefbbe, args=lion.qa.lab.tlv.redhat.com:/fastpass/gickowic/nfs1), log id: 5def745a >2013-03-12 16:21:25,670 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 16:21:25,678 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:21:25,678 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:21:27,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8702-1) [6405acd7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=361, mMessage=Storage domain is not empty - requires cleaning: ('lion.qa.lab.tlv.redhat.com:/fastpass/gickowic/nfs1',)]] >2013-03-12 16:21:27,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (ajp-/127.0.0.1:8702-1) [6405acd7] HostName = cheetah01.scl.lab.tlv.redhat.com >2013-03-12 16:21:27,206 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (ajp-/127.0.0.1:8702-1) [6405acd7] Command CreateStorageDomainVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed in vdscommand to CreateStorageDomainVDS, error = Storage domain is not empty - requires cleaning: ('lion.qa.lab.tlv.redhat.com:/fastpass/gickowic/nfs1',) >2013-03-12 16:21:27,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [6405acd7] FINISH, CreateStorageDomainVDSCommand, log id: 5def745a >2013-03-12 16:21:27,206 ERROR [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [6405acd7] Command org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed in vdscommand to CreateStorageDomainVDS, error = Storage domain is not empty - requires cleaning: ('lion.qa.lab.tlv.redhat.com:/fastpass/gickowic/nfs1',) >2013-03-12 16:21:27,210 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [6405acd7] Command [id=45a4abee-3866-4ab7-b779-0f03d5e6a538]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainDynamic; snapshot: 27a5cb52-099f-4b3a-aa29-ae9bc623d6fc. >2013-03-12 16:21:27,219 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [6405acd7] Command [id=45a4abee-3866-4ab7-b779-0f03d5e6a538]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: 27a5cb52-099f-4b3a-aa29-ae9bc623d6fc. >2013-03-12 16:21:27,235 ERROR [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [6405acd7] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand. >2013-03-12 16:21:27,359 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [12f05abe] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2013-03-12 16:21:27,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [12f05abe] START, DisconnectStorageServerVDSCommand(HostName = cheetah01.scl.lab.tlv.redhat.com, HostId = 7ca749d9-3a07-4d55-ab0e-43bb37fafcc1, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: lion.qa.lab.tlv.redhat.com:/fastpass/gickowic/nfs1, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 455c6f6c >2013-03-12 16:21:33,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [12f05abe] FINISH, DisconnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 455c6f6c >2013-03-12 16:22:16,298 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 16:22:16,306 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:22:16,307 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:22:25,666 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 16:22:25,674 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:22:25,674 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:23:16,301 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 16:23:16,308 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:23:16,308 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:23:25,662 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 16:23:25,669 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:23:25,670 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:24:16,303 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 16:24:16,310 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:24:16,310 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:24:25,662 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 16:24:25,670 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:24:25,670 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:25:16,305 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 16:25:16,314 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:25:16,314 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:25:25,661 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 16:25:25,668 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:25:25,668 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:26:16,307 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 16:26:16,314 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:26:16,314 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:26:25,651 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 16:26:25,658 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:26:25,659 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:27:16,308 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 16:27:16,326 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:27:16,326 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:27:25,651 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 16:27:25,658 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:27:25,659 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:28:16,310 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:28:16,317 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:28:16,318 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:28:25,649 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:28:25,654 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:28:25,655 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:29:16,312 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:29:16,318 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:29:16,318 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:29:25,648 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 16:29:25,655 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:29:25,655 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:30:16,314 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 16:30:16,323 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:30:16,323 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:30:25,647 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 16:30:25,654 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:30:25,654 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:30:33,946 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (QuartzScheduler_Worker-29) [5ea98a1a] Start refreshing all users data >2013-03-12 16:30:35,167 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (QuartzScheduler_Worker-97) [44516e24] Attempting to update VMs/Templates Ovf. >2013-03-12 16:31:16,316 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:31:16,324 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:31:16,324 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:31:25,645 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 16:31:25,661 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:31:25,661 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:32:16,316 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 16:32:16,324 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:32:16,324 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:32:25,749 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:32:25,758 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:32:25,758 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:33:16,318 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 16:33:16,325 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:33:16,325 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:33:25,641 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 16:33:25,648 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:33:25,648 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:34:16,320 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 16:34:16,327 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:34:16,328 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:34:25,629 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 16:34:25,636 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:34:25,637 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:35:16,322 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:35:16,331 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:35:16,332 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:35:25,631 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 16:35:25,638 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:35:25,638 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:36:16,323 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 16:36:16,330 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:36:16,331 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:36:25,622 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:36:25,629 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:36:25,630 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:37:16,325 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 16:37:16,332 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:37:16,333 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:37:25,624 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 16:37:25,632 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:37:25,632 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:38:16,326 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:38:16,333 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:38:16,334 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:38:25,623 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 16:38:25,629 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:38:25,629 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:39:16,327 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:39:16,333 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:39:16,334 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:39:25,623 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:39:25,630 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:39:25,631 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:40:16,330 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:40:16,377 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:40:16,377 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:40:25,612 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 16:40:25,619 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:40:25,619 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:41:16,330 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:41:16,337 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:41:16,337 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:41:25,613 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:41:25,621 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:41:25,621 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:42:16,332 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 16:42:16,341 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:42:16,341 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:42:25,609 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:42:25,615 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:42:25,615 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:43:16,333 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 16:43:16,341 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:43:16,341 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:43:25,602 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:43:25,609 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:43:25,610 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:44:16,335 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 16:44:16,343 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:44:16,343 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:44:25,609 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:44:25,620 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:44:25,620 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:45:16,337 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 16:45:16,345 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:45:16,345 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:45:25,605 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 16:45:25,612 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:45:25,612 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:46:16,339 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 16:46:16,347 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:46:16,347 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:46:25,600 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 16:46:25,607 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:46:25,608 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:47:16,341 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 16:47:16,348 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:47:16,348 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:47:25,600 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:47:25,608 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:47:25,608 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:48:16,341 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:48:16,348 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:48:16,349 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:48:25,594 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:48:25,605 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:48:25,605 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:49:16,343 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:49:16,350 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:49:16,351 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:49:25,593 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:49:25,601 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:49:25,601 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:50:16,344 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:50:16,352 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:50:16,352 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:50:25,589 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2013-03-12 16:50:25,596 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:50:25,596 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-2) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:51:16,346 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:51:16,354 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:51:16,354 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:51:25,586 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:51:25,593 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:51:25,594 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:52:16,349 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:52:16,356 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:52:16,356 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:52:25,580 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:52:25,588 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:52:25,588 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:53:16,348 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:53:16,356 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:53:16,356 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:53:25,590 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 16:53:25,597 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:53:25,598 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:54:16,350 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 16:54:16,355 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:54:16,355 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:54:27,505 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 16:54:27,510 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:54:27,510 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:55:16,352 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:55:16,360 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:55:16,361 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:55:27,220 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 16:55:27,225 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:55:27,225 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:56:16,353 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 16:56:16,360 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:56:16,360 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:56:27,211 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 16:56:27,218 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:56:27,218 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:57:16,354 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 16:57:16,362 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:57:16,363 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:57:27,214 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 16:57:27,221 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:57:27,221 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:58:16,357 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 16:58:16,364 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:58:16,364 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:58:27,217 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 16:58:27,224 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:58:27,225 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:59:16,358 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 16:59:16,364 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:59:16,365 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 16:59:27,218 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 16:59:27,225 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 16:59:27,225 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:00:16,361 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 17:00:16,371 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:00:16,371 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:00:27,221 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 17:00:27,228 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:00:27,228 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:01:16,363 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 17:01:16,370 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:01:16,370 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:01:27,224 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 17:01:27,230 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:01:27,231 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:02:16,367 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 17:02:16,374 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:02:16,374 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:02:27,233 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 17:02:27,244 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:02:27,244 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:03:16,366 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 17:03:16,373 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:03:16,374 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:03:27,229 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 17:03:27,235 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:03:27,235 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:04:16,368 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 17:04:16,375 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:04:16,376 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:04:27,229 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 17:04:27,236 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:04:27,236 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:05:16,369 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 17:05:16,377 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:05:16,378 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:05:27,231 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 17:05:27,238 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:05:27,238 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:06:16,371 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 17:06:16,379 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:06:16,379 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:06:27,232 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 17:06:27,240 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:06:27,240 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:07:16,373 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginUserCommand internal: false. >2013-03-12 17:07:16,380 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:07:16,380 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-6) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:07:27,233 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2013-03-12 17:07:27,242 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:07:27,242 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:08:16,374 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2013-03-12 17:08:16,381 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:08:16,381 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-11) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:08:27,242 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 17:08:27,249 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:08:27,250 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:09:16,376 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginUserCommand internal: false. >2013-03-12 17:09:16,383 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:09:16,384 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-9) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:09:27,244 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 17:09:27,251 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:09:27,251 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:10:16,378 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2013-03-12 17:10:16,384 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:10:16,384 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-5) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:10:27,245 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2013-03-12 17:10:27,252 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:10:27,253 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-4) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:11:16,380 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2013-03-12 17:11:16,387 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:11:16,387 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-3) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:12:16,403 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2013-03-12 17:12:16,413 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:12:16,413 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-1) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version >2013-03-12 17:13:16,384 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2013-03-12 17:13:16,391 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version >2013-03-12 17:13:16,391 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp-/127.0.0.1:8702-10) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=709932">View the attachment on a separate page</a>.</b>
View Attachment As Raw
Actions:
View
Attachments on
bug 921462
:
709932
|
711878